tiktoken

tiktoken is an open source byte pair encoding (BPE) tokenizer library released by OpenAI in December 2022 that converts text into the integer token sequences its language models read and write. OpenAI describes the project in a single line at the top of its README: "tiktoken is a fast BPE tokeniser for use with OpenAI's models."^[1] Written in Rust with Python bindings via PyO3, it ships the exact merge tables behind models including GPT-3.5, GPT-4, GPT-4o, GPT-4.1, and GPT-5, and is the implementation OpenAI's own cookbook recommends for counting tokens.^[1][2] The package distributes four main production encodings (r50k_base, p50k_base, cl100k_base, and o200k_base), the o200k_harmony variant used by the open-weight gpt-oss models, plus the legacy gpt2 and p50k_edit variants, and its README advertises a 3 to 6 times speedup over a comparable open source tokenizer when tokenizing 1 GB of text.^[1] tiktoken has become a de facto reference for measuring token counts in third party tooling: by November 2026 the PyPI package was being downloaded about 179 million times per month, with LangChain's OpenAI integration counting tokens with tiktoken and LlamaIndex using cl100k_base as its default token counter.^[31][4][32]

What is tiktoken?

tiktoken is a tokenizer, the component that sits between human-readable text and a language model. A transformer model does not operate on characters or words; it operates on integers drawn from a fixed vocabulary, and tiktoken is the function that maps a string to that list of integers (encoding) and back (decoding). Because OpenAI bills its API by the token and every model has a finite context window, tiktoken is also the standard tool developers use to predict request cost and to verify that a prompt fits before they send it.^[1][2]

The library is deliberately narrow. It does not train new vocabularies, normalize text, or support providers other than OpenAI; it ships pretrained OpenAI merge tables and exposes a small encode/decode surface around them. That focus is what lets it guarantee byte-identical output with OpenAI's production tokenizers, which is the property that makes it citable as ground truth for OpenAI token counts.^[1][2]

When was tiktoken released?

tiktoken's first PyPI release, version 0.1.1, was uploaded on 15 December 2022.^[6] The accompanying Hacker News thread, posted on 16 December 2022, documents the community's earliest reactions; users immediately noted that the new cl100k_base encoding doubled the vocabulary of the GPT-2 tokenizer from roughly 50,000 tokens to about 100,000 and gave special attention to numeric sequences.^[7] The repository's LICENSE file dates copyright to 2022 and credits OpenAI alongside individual contributor Shantanu Jain, the project's principal maintainer.^[8]

The library was developed alongside OpenAI's transition from the older byte-level BPE tokenizers shipped with GPT-2 and GPT-3 to a denser encoding for the GPT-3.5 and GPT-4 generations. Before tiktoken, many developers tokenized OpenAI inputs using the GPT2TokenizerFast class from Hugging Face's transformers library (backed by its tokenizers library), which reproduces the GPT-2 tokenizer published in February 2019.^[9] tiktoken provided a faster alternative tailored to the new vocabularies and shipped the token ranks as base64-encoded .tiktoken text files that are loaded at runtime.

Subsequent releases tracked OpenAI's model launches. Version 0.7.0 was published on 13 May 2024, the same day OpenAI announced GPT-4o and introduced the o200k_base encoding with a vocabulary of roughly 200,000 tokens.^[10][11] Version 0.8.0 followed on 3 October 2024 and added support for the o1 reasoning family, version 0.9.0 was released on 14 February 2025, version 0.13.0 was published on 15 May 2026, and version 0.14.0 (the current release as of November 2026) followed on 17 August 2026.^[10][3] Throughout this period the project remained MIT licensed and accepted contributions from outside OpenAI through GitHub pull requests, including community submissions that mapped new public model names (such as gpt-5) to the appropriate encoding.^[12]

Andrej Karpathy's February 2024 video lecture "Let's build the GPT Tokenizer" gave tiktoken a wider pedagogical audience by walking through the exact algorithm the library implements; the companion minbpe repository explicitly references tiktoken as the production target that the educational code mirrors.^[13][14]

How does tiktoken work?

tiktoken is structured as a small Rust crate with thin Python bindings. The hot path lives in the Rust module _tiktoken, exposed to Python via the PyO3 framework; the setup.py builds it with setuptools-rust, and the PyO3 version is pinned in Cargo.toml.^[1][15] The Python package (tiktoken/) contains the user-facing Encoding class, the lookup tables in model.py, and the tiktoken_ext plugin namespace under which encoding factories are registered.^[15]

Tokenization in tiktoken follows the standard byte-level BPE pipeline that traces back to Rico Sennrich, Barry Haddow, and Alexandra Birch's 2016 ACL paper on subword units for neural machine translation, which itself adapted the byte pair encoding compression algorithm published by Philip Gage in 1994.^[16] In tiktoken's variant, text is first split into chunks by a regular expression, and each chunk is then converted to UTF-8 bytes; the regex that captures contractions, runs of letters, runs of digits, runs of punctuation, and various whitespace classes. The Rust core then iteratively merges adjacent byte pairs according to a fixed merge ranking until no further merges from the encoding's vocabulary apply.^[1][17]

The regex split is close in spirit to what GPT-2 used but is compiled with the fancy_regex crate, using Unicode property escapes. The cl100k_base pretokenization pattern, for example, is:

'(?i:[sdmt]|ll|ve|re)|[^\r\n\p{L}\p{N}]?+\p{L}++|\p{N}{1,3}+| ?[^\s\p{L}\p{N}]++[\r\n]*+|\s++$|\s*[\r\n]|\s+(?!\S)|\s

This pattern matches English contractions case-insensitively, runs of Unicode letters, numeric groups of up to three digits, runs of non-letter non-digit punctuation, and trailing or interleaved whitespace.^[17] The o200k_base pattern, introduced for GPT-4o, generalizes the letter classes to include modifier letters and combining marks (\p{Lm}, \p{Lo}, and \p{M}) so that scripts like Devanagari, Tamil, and Arabic split into fewer pieces before the BPE merges run.^[18][19] The pattern also splits words on case (an uppercase run followed by lowercase letters) and attaches the same contraction suffixes to the preceding word instead of splitting them off.^[17]

Special tokens such as <|endoftext|>, the FIM fill-in-the-middle sentinels (<|fim_prefix|>, <|fim_middle|>, <|fim_suffix|>), and <|endofprompt|> are stored at fixed integer IDs above the BPE vocabulary. For cl100k_base, <|endoftext|> is token 100257 while the FIM tokens occupy 100258 through 100260; for o200k_base, <|endoftext|> is 199999 and <|endofprompt|> is 200018.^[17] tiktoken requires callers to opt into special tokens by listing them in the allowed_special argument of encode, which prevents accidental injection of FIM or end-of-text markers from untrusted user input.^[1]

A tiktoken_ext.openai_public plugin registers the four production encodings by name and points at .tiktoken rank files hosted on OpenAI's public Azure blob storage (openaipublic.blob.core.windows.net); it also registers gpt2, p50k_edit, and o200k_harmony. The first call to tiktoken.get_encoding("cl100k_base") downloads the rank file and caches it in a data-gym-cache folder under the system temporary directory (or a directory chosen via TIKTOKEN_CACHE_DIR), so subsequent calls on the same machine are offline while that cache survives.^[1][15]

The Rust core is exposed to Python through the CoreBPE class, which receives a Python dictionary mapping byte sequences to token integers, a separate dictionary of special tokens, and the precompiled regex string. The constructor builds an internal HashMap<Vec<u8>, u32> for the merge table and a reverse HashMap<u32, Vec<u8>> for the decoder, then compiles the regex with the fancy_regex crate (which supports the lookahead used in every encoding's split pattern) for use during encoding.^[1][15] The Rust encode and encode_ordinary methods on CoreBPE run without holding the Python global interpreter lock, which is what allows encode_batch to scale across threads when called from a Python thread pool.^[1]

Memory usage is dominated by the merge dictionary. The cl100k_base rank file is about 1.7 MB and the o200k_base file about 3.6 MB of base64 text; both are parsed fully into in-memory dictionaries at first use.^[17]

Which encodings does tiktoken use?

The four encodings shipped in the library correspond to four generations of OpenAI text models. The table below summarizes the mappings as of the tiktoken/model.py source on the main branch in May 2026.^[15][17]

The r50k_base encoding is byte-identical to the public GPT-2 tokenizer published by OpenAI in 2019: 256 base bytes, 50,000 learned merges, and a single <|endoftext|> token at index 50256.^[9] p50k_base extends this with merges learned on Codex's enlarged corpus (predominantly source code) and reserves several extra IDs for whitespace-only merges that improve indentation handling.^[20]

cl100k_base, first used by the text-embedding-ada-002 embedding model in December 2022 and then by the ChatGPT API's gpt-3.5-turbo (March 2023) and GPT-4 (14 March 2023), roughly doubles the vocabulary and adds first-class support for FIM completion. The OpenAI Cookbook recipe "How to count tokens with tiktoken" lists cl100k_base as the canonical encoding for gpt-4, gpt-4-turbo, gpt-3.5-turbo, and all three current embedding endpoints (text-embedding-ada-002, text-embedding-3-small, text-embedding-3-large).^[2]

o200k_base doubles the vocabulary again to roughly 200,000 entries. OpenAI shipped this encoding with GPT-4o on 13 May 2024 and uses it for every subsequent flagship model, including the o-series reasoning models (o1, o3, o3-mini, o4-mini) and the GPT-4.1 and GPT-5 families.^[11][15] OpenAI's MODEL_PREFIX_TO_ENCODING table uses prefix matching, so any model whose name starts with gpt-5, gpt-4.1-, gpt-4.5-, gpt-4o-, o1-, o3-, or o4-mini- automatically resolves to o200k_base (and gpt-oss- names to o200k_harmony) without requiring a new release of tiktoken every time OpenAI ships a versioned model.^[15]

How do you use tiktoken in Python?

The library exposes a compact Python API. Two factory functions create Encoding objects, and the resulting objects expose encode and decode methods that round-trip text and integers.^[1][2]

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
tokens = enc.encode("tiktoken is great!")
text = enc.decode(tokens)

enc4o = tiktoken.encoding_for_model("gpt-4o-mini")
print(len(enc4o.encode("Hello world")))

tiktoken.get_encoding(name) returns the encoding by explicit name, while tiktoken.encoding_for_model(model) looks the model up in the MODEL_TO_ENCODING table (or applies prefix matching) and returns the corresponding encoding. The OpenAI Cookbook documents the canonical pattern for token counting as len(encoding.encode(string)).^[2]

Beyond encode and decode, the Encoding class exposes encode_batch for parallel processing of multiple strings, encode_with_unstable for tokenizing input that may be a partial prefix of a larger string (useful when streaming completions), and decode_single_token_bytes for inspecting individual token bytes. The name, eot_token, and n_vocab attributes describe the encoding statically.^[1] Special tokens are disallowed by default; callers must pass allowed_special={"<|endoftext|>"} (or "all") to permit them, which guards against injecting end-of-text markers from user data.^[1]

The Python package depends only on regex for the pretokenization pattern and on requests for the one-time blob download; blobfile is an optional dependency used when applications need to load merge files from cloud object stores.^[3] The Rust core ships as a manylinux wheel for x86_64 and aarch64 Linux, plus separate x86_64 and arm64 wheels for macOS, musllinux wheels, and a Windows x86_64 wheel, so most users install with pip install tiktoken and never invoke cargo.^[3]

Two further utilities round out the API. tiktoken.list_encoding_names() returns the names of all registered encodings, including those contributed by plugins, and Encoding.encode_ordinary skips the special token check entirely for performance-sensitive paths where the caller knows the input contains no chat or FIM markers.

For applications that need to construct custom encodings (for example, while training a derivative model on a private vocabulary), tiktoken's Encoding constructor accepts the merge dictionary, the special tokens, the pretokenization regex, and an optional explicit vocabulary size directly. The library deliberately does not ship a training routine, so users wanting to learn new merges typically pair tiktoken.Encoding with an external trainer such as Karpathy's minbpe or Hugging Face's BpeTrainer, then load the resulting merges into a fresh Encoding for inference.^[14]

How fast is tiktoken?

The repository's README claims that "tiktoken is between 3-6x faster than a comparable open source tokeniser," referring to a benchmark that tokenizes 1 GB of text with the GPT-2 tokeniser against Hugging Face's GPT2TokenizerFast (measured with tokenizers==0.13.2, transformers==4.24.0, and tiktoken==0.2.0).^[1] An independent benchmark on machinelearningplus.com found a smaller 2 to 3 times speedup for tiktoken on single strings, and noted that Hugging Face gets close for batch jobs because its encode_batch runs across Rust threads.^[21]

The speed advantage comes from two design choices. First, tiktoken's pretokenization regex is precompiled once with the fancy_regex crate, and the Rust core avoids allocating intermediate Python objects during the inner merge loop.^[1] Second, the BPE merges are stored as a flat HashMap<Vec<u8>, u32> keyed by byte sequences rather than the trie-based approach used in some earlier implementations, which trades a small amount of memory for cache-friendly lookups.^[1]

Several newer Rust tokenizers have since outperformed tiktoken on specific benchmarks. The bpe crate's README reports its algorithm as about 10 times faster than Hugging Face's BPE for standard encoding and counting, and similar to tiktoken in typical cases but with linear rather than quadratic scaling on worst-case inputs.^[22] The rs-bpe project published comparisons in 2025 claiming it consistently outperforms both tiktoken and Hugging Face's tokenizers in its benchmarks.^[23] These results sit alongside tiktoken rather than displacing it: tiktoken's value is that it ships the exact OpenAI merge tables under the MIT license and is the implementation the official cookbook recommends, so applications targeting OpenAI models continue to use it for correctness even when raw throughput matters less than fidelity.^[2]

How does tiktoken handle non-English text?

The o200k_base encoding's principal practical change over cl100k_base is improved compression of non-English text. OpenAI's GPT-4o launch post reported large reductions in token counts for Indic, Arabic, Chinese, and other non-Latin scripts, from 1.1 times fewer tokens for an English sample to 4.4 times fewer for Gujarati.^[11] A later analysis on njkumar.com cited a Tamil example where the sentence "நீ எப்படி இருக்கிறாய்? நான் நல்லா இருக்கேன்" was tokenized as 68 tokens with cl100k_base but only 21 with o200k_base, a 3.2 times compression improvement.^[19] A comparison on aipmguru.substack.com measured the cumulative change from GPT-3's tokenizer to the current o200k_base: a 17 percent reduction in tokens for an English sample and an 86 percent reduction for a Gujarati paragraph.^[24] In OpenAI's own launch samples, an Arabic sentence fell from 53 to 26 tokens and a Chinese sentence from 34 to 24.^[11]

Two factors drive the improvement. First, the larger merge table can absorb full common phrases as single tokens in scripts where cl100k_base had to fall back to byte-level encoding. Second, the pretokenization regex was rewritten to use the Unicode general categories \p{Lo} (other letters), \p{Lm} (modifier letters), and \p{M} (combining marks), which prevents the splitter from cutting between a base character and its combining diacritic before the BPE merges have a chance to operate.^[17][19]

The compression improvement matters for users in two ways. First, because OpenAI bills its API by token, the same source text in Hindi or Korean costs less to send to GPT-4o than to GPT-4 turbo. Second, the model can fit more content in its context window when the encoding is denser. The trade-off is that a larger vocabulary enlarges the model's embedding table and the softmax over the output distribution.

The GPT-4o launch post presented the new tokenizer in a 'Language tokenization' section with per-language samples.^[11] In the source, o200k_base has 199,998 BPE tokens plus two special tokens,^[17] and the o200k_harmony variant shipped with the open-weight gpt-oss models in August 2025 extends it to 201,088 IDs with special tokens for chat formatting and role-based prompting.^[25]

How does tiktoken compare to other tokenizers?

The two principal alternatives to tiktoken in the open ecosystem are Hugging Face's tokenizers library and Google's SentencePiece. All three implement the BPE family but differ in scope, training capability, and licensing.

Hugging Face tokenizers, also written in Rust with Python bindings, is a much broader toolkit. It supports BPE, WordPiece, and Unigram models, exposes a PreTokenizer and Normalizer configuration system, and ships a train_from_iterator method that fits new tokenizers from corpora. By contrast, tiktoken is inference only: it ships pretrained OpenAI merge tables and offers no training API.^[1][26] In raw token counting, tiktoken is faster on single strings, while Hugging Face gets close once its encode_batch parallelizes across Rust threads.^[21]

SentencePiece, published by Taku Kudo and John Richardson at Google in 2018, takes a different architectural stance: it treats input as a sequence of Unicode characters and trains directly on unprocessed text, which makes it language agnostic and removes the need for a separate pretokenization regex. SentencePiece supports both BPE and Unigram language model tokenization and has been used by models such as T5 and the first two LLaMA generations.^[27] SentencePiece's BPE differs from tiktoken's primarily because it escapes whitespace with a fixed U+2581 meta symbol rather than relying on bytes for whitespace; this makes its outputs incompatible with tiktoken's merge tables, but the algorithmic complexity is similar.^[27]

A third class of implementation, often grouped under the loose umbrella "GPT2-style" tokenizers, includes pure JavaScript ports (such as gpt-tokenizer and js-tiktoken), .NET ports (tryAGI/Tiktoken), and Go and R bindings that reuse the official cl100k_base and o200k_base merge files. These ports exist because OpenAI's official package only targets Python. None of these ports trains new tokenizers either: they exist purely to count or encode tokens against OpenAI's vocabularies in non-Python environments.^[28]

Karpathy's minbpe and rustbpe projects exist explicitly as educational reference implementations. minbpe reproduces tiktoken's encode path in roughly 200 lines of Python and adds a small training loop; rustbpe extends the Rust path with a train method that tiktoken lacks. Both projects target byte-compatible output with tiktoken's cl100k_base for any input.^[14]

Who uses tiktoken?

tiktoken sits at the foundation of nearly every Python tool that interacts with OpenAI's text APIs. pypistats reported about 179 million downloads in the month to early November 2026, with about 8.6 million on the most recent day, ranking it among the most downloaded AI-adjacent libraries on the index.^[31]

LangChain's OpenAI chat integration (langchain_openai) uses tiktoken inside its get_num_tokens_from_messages implementation, resolving each model's encoding and falling back to cl100k_base or o200k_base.^[32] LlamaIndex sets cl100k_base as its global default tokenizer via the Settings.tokenizer attribute; the project's documentation specifically calls out the dependency on tiktoken to match GPT-3.5 turbo's default behavior.^[4] LiteLLM, an aggregation layer that exposes a single API across multiple providers, falls back to tiktoken when no provider-specific tokenizer is available.^[5]

Third party token counters for non-OpenAI models often use tiktoken as a rough approximation. Anthropic's official position is that Claude requires its own client.messages.count_tokens endpoint for accurate counts, and its older anthropic-tokenizer-typescript package is, by its own README, no longer accurate for Claude 3 and later models.^[29][30]

tiktoken is also the canonical reference implementation cited by academic papers when reporting token-based metrics for OpenAI evaluation. Papers studying multilingual token compression, prompt cost analysis, and prompt injection routinely report token counts using tiktoken's cl100k_base or o200k_base and link the exact library version they used for reproducibility.^[19][31]

The library has been ported into more than a dozen non-Python ecosystems. The tiktoken-go package provides Go bindings, gpt-tokenizer and js-tiktoken cover Node.js and the browser, tryAGI/Tiktoken covers .NET, and rtiktoken provides an R interface that wraps the official Rust core through extendr. Most of these projects bundle the official merge tables and reach byte-identical output with the Python package for the supported encodings, which means their token counts can be used as the ground truth for OpenAI billing in JavaScript, Go, and .NET applications without round-tripping calls to a Python service.^[28]

What are tiktoken's limitations?

The library exposes only inference. Users cannot fit new BPE vocabularies through tiktoken itself; producing a new encoding requires an external trainer and then constructing a tiktoken.Encoding around the resulting merges. The README instead points readers to its tiktoken._educational submodule, which includes a small toy trainer for learning how BPE works, and Karpathy's minbpe offers a fuller training implementation.^[1][14]

Special token handling is conservative by design. Calling encode("<|endoftext|>") raises a ValueError unless the caller passes allowed_special={"<|endoftext|>"} or sets allowed_special="all". This prevents prompt injection through user-supplied content that contains the literal special token string, but it can surprise developers who unwittingly include the token in their templates. The opposite mode, disallowed_special=(), opts every special token back into normal encoding paths and is sometimes used to round-trip arbitrary text without errors.^[1]

tiktoken's merge files are downloaded from blob storage on first use, which means the library does not work in air-gapped environments without prior preparation. The usual workaround, based on the cache logic in tiktoken/load.py, is to populate a cache directory in a connected environment by setting TIKTOKEN_CACHE_DIR, copy that directory to the target machine, and set TIKTOKEN_CACHE_DIR there too. Several community ports (such as js-tiktoken and the .NET Tiktoken package) bundle the merge tables directly to avoid this problem.^[1][7][28]

The library targets OpenAI's vocabularies and matches them exactly, but it offers no tokenizers for other providers. Counts produced by tiktoken.get_encoding("cl100k_base") for a Claude or LLaMA input are approximations that drift from the actual model's tokenizer by a few percent on English and considerably more on multilingual text. Anthropic's documentation directs developers to its own client.messages.count_tokens endpoint for Claude, and the transformers library's LlamaTokenizer is the canonical counter for Meta's models.^[29][30]

Finally, because the merge tables are byte-level, tiktoken cannot guarantee that arbitrary sequences of tokens decode to valid UTF-8. Individual tokens may be partial UTF-8 byte sequences that only become valid when concatenated with neighbors; decode_single_token_bytes returns raw bytes precisely so that callers handling streaming output can buffer until a complete code point arrives. This behaviour is documented in the API docstrings in tiktoken/core.py.^[1]

ELI5: What does tiktoken actually do?

Language models do not read words; they read numbers. Before GPT-4 can process the sentence "tiktoken is great!", that text has to be chopped into pieces called tokens and each piece swapped for a number from a fixed list. tiktoken is the tool that does the chopping and the swapping. It uses a method called byte pair encoding, which starts from raw bytes and repeatedly glues together the most common neighboring pairs, so frequent chunks like " the" or "ing" become single tokens while rare words get split into several pieces. The number of tokens is exactly what OpenAI charges you for and exactly what fills up a model's memory limit, so people use tiktoken to answer two everyday questions: how much will this prompt cost, and will it fit?

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22OpenAI, "openai/tiktoken: tiktoken is a fast BPE tokeniser for use with OpenAI's models", GitHub, 2026-05-15. github.com/...tiktoken. Accessed 2026-05-26.
2. ^1 ^2 ^3 ^4 ^5 ^6 ^7OpenAI, "How to count tokens with tiktoken", OpenAI Cookbook. developers.openai.com/...ount_tokens_with_tiktoken. Accessed 2026-05-26.
3. ^1 ^2 ^3tiktoken project, "tiktoken 0.13.0", PyPI, 2026-05-15. pypi.org/...tiktoken. Accessed 2026-05-26.
4. ^1 ^2LlamaIndex, "Using LLMs", LlamaIndex documentation. docs.llamaindex.ai/...llms. Accessed 2026-05-26.
5. ^BerriAI, "Completion Token Usage and Cost", LiteLLM documentation. docs.litellm.ai/...token_usage. Accessed 2026-05-26.
6. ^tiktoken project, "tiktoken 0.1.1", PyPI, 2022-12-15. pypi.org/...0.1.1. Accessed 2026-05-26.
7. ^1 ^2Hacker News, "Tiktoken: OpenAI's Tokenizer", Y Combinator, 2022-12-16. news.ycombinator.com/item. Accessed 2026-05-26.
8. ^OpenAI, "tiktoken/LICENSE", GitHub, 2022. github.com/...LICENSE. Accessed 2026-05-26.
9. ^1 ^2Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever, "Language Models are Unsupervised Multitask Learners", OpenAI, 2019-02-14. cdn.openai.com/...upervised_multitask_learners.pdf. Accessed 2026-05-26.
10. ^1 ^2OpenAI, "Releases: openai/tiktoken", GitHub. github.com/...releases. Accessed 2026-05-26.
11. ^1 ^2 ^3 ^4 ^5OpenAI, "Hello GPT-4o", OpenAI blog, 2024-05-13. openai.com/...hello-gpt-4o. Accessed 2026-05-26.
12. ^rjarun8235, "Add GPT-5 model support with o200k_base encoding", openai/tiktoken pull request 440, GitHub. github.com/...440. Accessed 2026-05-26.
13. ^Andrej Karpathy, "Let's build the GPT Tokenizer", YouTube, 2024-02-20. youtube.com/watch. Accessed 2026-05-26.
14. ^1 ^2 ^3 ^4Andrej Karpathy, "karpathy/minbpe: Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization", GitHub, 2024-02-16. github.com/...minbpe. Accessed 2026-05-26.
15. ^1 ^2 ^3 ^4 ^5 ^6 ^7OpenAI, "tiktoken/tiktoken/model.py", GitHub. github.com/...model.py. Accessed 2026-05-26.
16. ^Rico Sennrich, Barry Haddow, and Alexandra Birch, "Neural Machine Translation of Rare Words with Subword Units", Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), 2016-08-07. aclanthology.org/P16-1162. Accessed 2026-05-26.
17. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8OpenAI, "tiktoken/tiktoken_ext/openai_public.py", GitHub. github.com/...openai_public.py. Accessed 2026-05-26.
18. ^Various contributors, "o200k_base pretokenizer regex error?", openai/tiktoken issue 298, GitHub. github.com/...298. Accessed 2026-05-26.
19. ^1 ^2 ^3 ^4NJ Kumar, "Multilingual token compression in GPT-o family models", njkumar.com, 2024-10-07. njkumar.com/gpt-o-multilingual-token-compression. Accessed 2026-05-26.
20. ^OpenAI, "Models", OpenAI Platform documentation. platform.openai.com/...models. Accessed 2026-05-26.
21. ^1 ^2MachineLearningPlus, "tiktoken vs HuggingFace Tokenizers: Benchmark Guide", 2024. machinelearningplus.com/...-huggingface-tokenizers. Accessed 2026-05-26.
22. ^Github user contribution, "bpe", crates.io. crates.io/...bpe. Accessed 2026-05-26.
23. ^gweidart, "rs-bpe outperforms tiktoken and tokenizers", DEV Community, 2025-03-18. dev.to/...bpe-outperforms-tiktoken-tokenizers-2h3j. Accessed 2026-05-26.
24. ^AIPM Guru, "The Invisible Upgrade: How Tokenization Quietly Got Better (And Why Your AI Costs Dropped)", Substack. aipmguru.substack.com/...-upgrade-how-tokenization. Accessed 2026-05-26.
25. ^Modal, "What is o200k Harmony? OpenAI's latest edition to their tiktoken tokenizer library", Modal blog, 2025. modal.com/...what-is-o200k-harmony. Accessed 2026-05-26.
26. ^Hugging Face, "huggingface/tokenizers", GitHub. github.com/...tokenizers. Accessed 2026-05-26.
27. ^1 ^2Taku Kudo and John Richardson, "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing", arXiv:1808.06226, 2018-08-19. arxiv.org/...1808.06226. Accessed 2026-05-26.
28. ^1 ^2 ^3tryAGI, "tryAGI/Tiktoken: High performance .NET BPE tokenizer", GitHub. github.com/...Tiktoken. Accessed 2026-05-26.
29. ^1 ^2Anthropic, "Token counting", Claude API docs. platform.claude.com/...token-counting. Accessed 2026-05-26.
30. ^1 ^2Anthropic, "anthropics/anthropic-tokenizer-typescript", GitHub. github.com/...anthropic-tokenizer-typescript. Accessed 2026-05-26.
31. ^1 ^2 ^3tiktoken project, "tiktoken download statistics", pypistats.org. pypistats.org/...tiktoken. Accessed 2026-11-05.
32. ^1 ^2LangChain, "langchain_openai/chat_models/base.py", GitHub. github.com/...base.py. Accessed 2026-11-05.