370
edits
Tokens: Difference between revisions
no edit summary
No edit summary |
|||
| Line 27: | Line 27: | ||
*[[Tiktoken]], a fast BPE tokenizer specifically for OpenAI models | *[[Tiktoken]], a fast BPE tokenizer specifically for OpenAI models | ||
*[[Transformers]] package for Python | *[[Transformers]] package for Python | ||
*[ | *[https://www.npmjs.com/package/gpt-3-encoder gpt-3-encoder package for node.js] | ||
==Token Limits== | ==Token Limits== | ||