370
edits
Tokens: Difference between revisions
→Tokenization Tools
No edit summary |
|||
| Line 25: | Line 25: | ||
*[https://platform.openai.com/tokenizer OpenAI's interactive Tokenizer tool] | *[https://platform.openai.com/tokenizer OpenAI's interactive Tokenizer tool] | ||
*[ | *[https://github.com/openai/tiktoken Tiktoken], a fast BPE tokenizer specifically for OpenAI models | ||
*[[Transformers]] package for Python | *[[Transformers]] package for Python | ||
*[https://www.npmjs.com/package/gpt-3-encoder gpt-3-encoder package for node.js] | *[https://www.npmjs.com/package/gpt-3-encoder gpt-3-encoder package for node.js] | ||