# SentencePiece

> Source: https://aiwiki.ai/wiki/sentencepiece
> Updated: 2026-06-22
> Categories: Machine Learning, Natural Language Processing, Open Source AI
> From AI Wiki (https://aiwiki.ai), a free encyclopedia of artificial intelligence. Quote with attribution.

**SentencePiece** is an open-source, language-agnostic [subword tokenization](/wiki/subword_tokenization) library and detokenizer developed at [Google](/wiki/google) and introduced by [Taku Kudo](/wiki/taku_kudo) and [John Richardson](/wiki/john_richardson) in their 2018 EMNLP system demonstration paper.[^1] It trains a subword [tokenization](/wiki/tokenization) model directly from raw [Unicode](/wiki/unicode) text, with no external pre-tokenization step, and supports two main segmentation algorithms: [byte-pair encoding](/wiki/byte_pair_encoding) (BPE) and a [unigram language model](/wiki/unigram_lm). The original paper describes it as "a language-independent subword tokenizer and detokenizer designed for Neural-based text processing" that forms "a purely end-to-end and language independent system."[^1] Released under the [Apache 2.0 license](/wiki/apache_2_0_license) and hosted at `github.com/google/sentencepiece`, it is downloaded more than 36 million times per month from PyPI, making it one of the most widely used dependencies in open natural language processing.[^3][^24]

SentencePiece has become a foundational piece of modern [natural language processing](/wiki/natural_language_processing) infrastructure. It powers the tokenizers used by [T5](/wiki/t5), [ALBERT](/wiki/albert), [XLNet](/wiki/xlnet), [mT5](/wiki/mt5), [mBART](/wiki/mbart), [NLLB](/wiki/nllb), [Marian NMT](/wiki/marian_nmt), [Llama](/wiki/llama) 1 and 2, [Mistral 7B](/wiki/mistral_7b), [Gemma](/wiki/gemma), [PaLM](/wiki/palm), and many [DeepSeek](/wiki/deepseek) and Qwen releases, among other large-scale [language models](/wiki/language_model).[^1][^3][^11][^17][^18] Its design choices also influenced a generation of fast Rust-based tokenizers in the [Hugging Face](/wiki/hugging_face) Tokenizers ecosystem, which can load SentencePiece model files directly.[^15]

## What problem did SentencePiece solve?

Most early subword tokenizers, including the influential BPE implementation released by Sennrich, Haddow, and Birch in 2016 for [neural machine translation](/wiki/nmt), expected pre-tokenized text as input.[^4] The standard pipeline for English or other space-separated languages typically involved a chain of tools such as the Moses tokenizer, which would split punctuation from words, normalize quote characters, and apply language-specific rules before any subword model saw the text. This worked acceptably for European languages but introduced several real problems.

First, languages without explicit word boundaries, including Chinese, Japanese, Thai, and many Southeast Asian scripts, do not have whitespace separating words. Pre-tokenization for these languages requires external segmenters such as MeCab for Japanese or the Stanford Word Segmenter for Chinese, each adding installation complexity and producing slightly different segmentations across versions.[^1] Second, end-to-end models that learn directly from raw text cannot easily integrate language-specific tokenizers without breaking the goal of language independence. Third, detokenization in this multistage pipeline is lossy because spacing and punctuation conventions cannot always be reconstructed from the token sequence alone.

A second motivation was engineering. Earlier subword toolkits relied on multiple Python scripts, shell pipelines, and ancillary lookup tables that were hard to package inside a production model server. The authors of SentencePiece wanted a tokenizer that ships as a single self-contained model file, can be loaded into any process without additional dictionaries or scripts, and can be reproduced bit-for-bit across machines.[^1][^3] The model file is encoded in [Protocol Buffers](/wiki/protocol_buffers) format and contains the vocabulary, the merge rules or unigram probabilities, the normalization rules, and bookkeeping metadata. Eliminating the external pre-tokenization step also made segmentation a pure function of the trained model file, which can be checked into source control and shipped along with the trained weights.[^1]

## Who created SentencePiece and when?

SentencePiece was developed primarily by Taku Kudo and John Richardson at Google in Tokyo, with the first public release on GitHub in 2017 and the academic publication appearing as a system demonstration at EMNLP 2018.[^1][^3] Taku Kudo had already been a well-known figure in Japanese natural language processing for more than a decade as the original author of MeCab, the most widely used Japanese morphological analyzer, and of CRF++, an early and influential conditional random field implementation. His prior work in Japanese segmentation strongly shaped the design choices that made SentencePiece language-agnostic from the start: the same researcher who had spent years building MeCab understood firsthand why depending on a language-specific external segmenter for multilingual text was a structural problem.[^1]

The unigram language model that became one of SentencePiece's two flagship algorithms was introduced by Kudo earlier in 2018 in a separate ACL paper on subword regularization, where it was used to provide stochastic alternative segmentations for training neural translation models.[^2] The SentencePiece library bundled this algorithm together with BPE in a single C++ implementation, with the explicit goal of making both methods available behind a common interface.

The system demonstration paper for the library itself, "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing," appeared in the proceedings of EMNLP 2018 system demonstrations and on arXiv as preprint 1808.06226.[^1] John Richardson co-authored the demonstration paper and contributed substantially to the library's engineering, including its Python and TensorFlow bindings.[^1][^3]

## Design principles

SentencePiece's design rests on a few deliberate principles that distinguish it from earlier subword tokenizers and that have been broadly adopted by its successors.[^1][^3] The GitHub repository sums up the goal directly: it provides "a language-independent pipeline that completely eliminates the need for language-specific pre- or post-processing."[^3]

The first principle is that the tokenizer must operate on raw text rather than pre-tokenized input. The library encodes the space character as the meta symbol `U+2581` (LOWER ONE EIGHTH BLOCK) and then trains its subword model on the raw Unicode text.[^1] Because the space symbol is part of the vocabulary and appears as a prefix on subword tokens that follow whitespace, detokenization is reduced to concatenating the pieces and replacing `U+2581` with a regular space. This eliminates the need for language-specific tokenizers and makes the model file the sole source of segmentation behavior.

The second principle is lossless detokenization. The library is designed so that for any input text, encoding followed by decoding produces output that is byte-for-byte identical to the input, subject only to the configured [Unicode normalization](/wiki/nfkc_normalization).[^1] This contrasts with earlier pipelines where punctuation splitting and whitespace handling discarded information that could not be recovered at decoding time.

The third principle is language independence. Any UTF-8 text corpus can be fed to the trainer without language-specific preprocessing or dictionaries, including text in scripts without word boundaries.[^1] This is essential for multilingual models that share a single tokenizer across many languages, including modern systems such as mT5 (101 languages) and NLLB (more than 200 languages).[^6][^9]

The fourth principle is self-containment. The trained model is a single Protocol Buffers serialized file containing all information needed to reproduce segmentation, with no external dictionaries or language-specific resources.[^3] A related design assumption, stated explicitly in the project README, is that "the vocabulary size is fixed prior to training," which is what allows a model team to budget an exact token count for an embedding table before any training begins.[^3] The fifth principle is determinism: given a fixed model file and fixed normalization, identical input strings always produce identical token sequences, across operating systems and across releases.[^3] This is a critical property for large-model training reproducibility, where any drift in the input tokenization would silently change loss values across machines.

## What algorithms does SentencePiece support?

SentencePiece supports four model types selected at training time via the `--model_type` flag: `bpe`, `unigram`, `word`, and `char`. The first two are the practically important options, the latter two exist mostly as baselines or for special cases.[^3]

### BPE mode

The BPE algorithm in SentencePiece is a faithful implementation of the original byte-pair encoding compression algorithm proposed by Philip Gage in 1994 and adapted to subword segmentation by Sennrich, Haddow, and Birch in 2016.[^4] Training proceeds bottom-up: the corpus is first split into individual Unicode characters, then the most frequent adjacent pair of symbols is merged into a new symbol, and the process repeats until the vocabulary reaches the target size. The ordered list of merges is stored in the model, and at inference time the same merge list is applied greedily to new input, guaranteeing deterministic segmentation.[^4]

SentencePiece's BPE differs from the original Sennrich implementation in two visible ways. First, it operates on raw text where whitespace has been mapped to `U+2581`, so merges can cross what used to be word boundaries if such merges are common in the training data. In practice this is rare because cross-word character combinations are not frequent enough to outrank within-word combinations during early merges.[^1] Second, the SentencePiece BPE encoder is implemented in C++ with a priority-queue-based merge step, substantially faster than the Python reference implementation that shipped with the original paper.[^3]

When `byte_fallback=True` is set, BPE training adds 256 byte-level tokens (one for each value `0x00` through `0xFF`) to the vocabulary so that any character that the trained merges cannot produce can still be represented as a sequence of UTF-8 bytes.[^3] This option is used by [Llama](/wiki/llama) 1 and 2 and by [Gemma](/wiki/gemma) to avoid out-of-vocabulary tokens entirely while keeping the merged vocabulary at a manageable size.[^10][^11]

### Unigram language model mode

The unigram subword model, introduced by Kudo in 2018,[^2] takes the opposite top-down approach. Training begins with a large seed vocabulary of candidate substrings drawn from the corpus, often built using suffix arrays or simple heuristics, and then iteratively prunes pieces whose removal causes the smallest decrease in the marginal likelihood of the data. The probability of a sentence under the unigram model is the product of the probabilities of the pieces in the chosen segmentation, marginalized over all valid segmentations using a forward-backward computation during training and a Viterbi search at inference time.[^2]

The unigram model has several attractive properties. Because it is probabilistic, multiple valid segmentations exist for any input, and the library can sample alternative segmentations from the posterior, which is the basis of [subword regularization](/wiki/subword_regularization).[^2] Vocabularies trained with the unigram method tend to contain fewer suffix-only or prefix-only fragments and more linguistically meaningful pieces such as common morphemes. Unigram training is more expensive than BPE training for the same target vocabulary size, but unigram models often perform slightly better in [BLEU](/wiki/bleu_score) evaluations on low-resource language pairs, especially when subword regularization is applied during training.[^2][^3]

#### Subword regularization

Subword regularization is the training-time technique that gives the unigram model its main practical advantage over greedy BPE.[^2] Instead of always feeding the model the single most likely segmentation of each input sentence, the training pipeline samples a different segmentation each epoch, drawn from the posterior over tokenizations under the trained unigram model. The model therefore sees the same content under many different surface forms, which has a regularizing effect similar to dropout on token sequences.

In the original ACL 2018 paper, Kudo reported BLEU improvements of roughly one to three points on low-resource and out-of-domain neural machine translation tasks.[^2] The technique requires no architectural changes to the downstream model. Two hyperparameters control the sampling: `nbest_size`, which truncates the posterior to the top n candidates, and `alpha`, the temperature applied to the truncated distribution.[^2] For BPE, an analogous technique called BPE-dropout was introduced by Provilkov, Emelianenko, and Voita in 2020;[^12] SentencePiece's `bpe_dropout` runtime parameter at encode time provides similar functionality natively.

### Word and character modes

The `word` model splits on whitespace only and uses each unique word as a token, capped at the requested vocabulary size with the rest going to a single unknown token. The `char` model uses each Unicode codepoint as its own token.[^3] Both options are mostly used for ablation studies and for very small models where subword granularity is not needed. They are also occasionally useful when a downstream model has a custom architecture for character or word inputs that does not require a subword vocabulary.

## Why does SentencePiece use the `▁` whitespace symbol?

The single most distinctive design choice of SentencePiece, and the one that the library is most often recognized by visually, is its replacement of the space character with the Unicode code point `U+2581`, displayed as `▁` (LOWER ONE EIGHTH BLOCK).[^1] Before training begins, every whitespace character in the input is mapped to this symbol; during inference, the same mapping is applied to incoming text. The marker therefore appears as a prefix on the first subword that follows any whitespace, making word boundaries explicit and learnable as part of the token vocabulary itself.

This choice has several practical consequences. Detokenization becomes trivial: the tokens are concatenated, and every `▁` is replaced by a regular space. The original text is recovered exactly, including punctuation handling and any number of consecutive spaces. There is no need for a separate detokenizer step that knows about language-specific punctuation rules. Token sequences also become directly inspectable in a way that older subword schemes did not allow, because the boundary information is visible in the tokens themselves.[^1]

The choice of `U+2581` rather than a more familiar symbol such as an underscore was made to ensure that the meta symbol does not collide with any character that might appear in the source text. The LOWER ONE EIGHTH BLOCK character is extremely rare in natural language text. When inspecting tokens, the symbol is often shown in terminals and editors with a small visible underscore-like glyph, which is why it is sometimes informally called "the underscore" in tutorials despite not being one. For models that need to start a sequence with a token that does not follow whitespace, SentencePiece's `add_dummy_prefix` flag inserts a leading `▁` automatically.[^3]

## How does SentencePiece train a tokenizer?

SentencePiece training takes a raw text corpus, a target vocabulary size, and a model type, and produces a single self-contained `.model` file along with a human-readable `.vocab` file that lists every piece and its score.[^3]

The trainer reads input either from plain text files passed via `--input` or from standard input. Each line of input is treated as one sentence; the library does not perform sentence splitting on its own. Lines longer than `--max_sentence_length` are skipped by default, and `--input_sentence_size` caps how many input sentences are used, important for very large corpora where training on every sentence is unnecessary.[^3]

The training pipeline first applies Unicode normalization, defaulting to NFKC normalization, which folds compatibility characters such as full-width digits and ligatures into their canonical forms.[^3] Whitespace is then collapsed and mapped to `▁`, control symbols and any user-defined symbols are reserved, and the corpus is passed to the chosen model trainer.

For BPE, the trainer counts adjacent symbol pairs across the corpus, repeatedly merges the most frequent pair into a new symbol, and updates the counts efficiently using a priority queue. For unigram, the trainer initializes a large candidate vocabulary, runs an expectation-maximization loop that estimates each piece's probability, and prunes the lowest-scoring pieces in batches until the vocabulary reaches the target size.[^2][^3] Training is multithreaded; on modern hardware, training a 32,000-piece BPE vocabulary on a few gigabytes of text typically takes minutes, while a comparable unigram vocabulary takes several times longer.

## API and language bindings

SentencePiece ships with a small set of command-line tools and bindings for several languages.[^3] The basic training, encoding, and decoding workflow looks as follows in shell.

```
spm_train --input=corpus.txt \
          --model_prefix=mymodel \
          --vocab_size=32000 \
          --model_type=bpe

spm_encode --model=mymodel.model < text.txt > tokens.txt

spm_decode --model=mymodel.model < tokens.txt > reconstructed.txt
```

The Python bindings, installed with `pip install sentencepiece`, expose the same operations through a class-based API.

```
import sentencepiece as spm

spm.SentencePieceTrainer.train(
    input='corpus.txt',
    model_prefix='mymodel',
    vocab_size=32000,
    model_type='bpe',
    byte_fallback=True
)

sp = spm.SentencePieceProcessor(model_file='mymodel.model')
pieces = sp.encode_as_pieces('Hello, world!')
ids = sp.encode_as_ids('Hello, world!')
text = sp.decode_pieces(pieces)
```

For the unigram model, the same code can be used with `model_type='unigram'`, and the runtime additionally exposes the `sample_encode_as_pieces` method, which draws a tokenization from the posterior. Setting `nbest_size` and `alpha` controls the temperature of the sampling and is the entry point to subword regularization during training of downstream models.[^2]

The library ships with bindings for C++, Python, Java, JavaScript via WebAssembly, and TensorFlow ops, with Rust and Go bindings available through community projects.[^3] The shipped Python wheels include precompiled binaries for Linux, macOS, and Windows on common architectures. The TensorFlow ops package, `tensorflow-text`, integrates SentencePiece as a native graph operation, which allows tokenization inside a TensorFlow graph without round-tripping to Python.[^3]

## Which models use SentencePiece?

SentencePiece tokenizers underpin a large fraction of widely used language models. The following table summarizes notable models, their tokenizer choice within the SentencePiece family, and approximate vocabulary size, based on each model's published configuration or paper.

| Model | Tokenizer type | Vocabulary size | Notes |
|---|---|---|---|
| [T5](/wiki/t5) | SentencePiece unigram | 32,000 | English-centric encoder-decoder, 2019[^5] |
| [mT5](/wiki/mt5) | SentencePiece unigram | 250,000 | 101 languages, 2020[^6] |
| [ALBERT](/wiki/albert) | SentencePiece unigram | 30,000 | Lite BERT, 2019[^7] |
| [XLNet](/wiki/xlnet) | SentencePiece | 32,000 | Autoregressive pretraining, 2019[^8] |
| [mBART](/wiki/mbart) | SentencePiece | 250,000 | Multilingual denoising autoencoder |
| [NLLB](/wiki/nllb) | SentencePiece | 256,000 | 200+ languages, 2022[^9] |
| [Marian NMT](/wiki/marian_nmt) | SentencePiece (default) | varies | Production MT framework |
| [Llama](/wiki/llama) 1 | SentencePiece BPE | 32,000 | byte_fallback enabled, 2023[^10] |
| [Llama](/wiki/llama) 2 | SentencePiece BPE | 32,000 | Same tokenizer family as Llama 1, 2023[^17] |
| [Mistral 7B](/wiki/mistral_7b) | SentencePiece BPE | 32,000 | byte_fallback enabled, 2023[^18] |
| [Gemma](/wiki/gemma) | SentencePiece | 256,000 | Subset of Gemini's tokenizer, 2024[^11] |
| [PaLM](/wiki/palm) | SentencePiece | 256,000 | 540B parameter dense model, 2022[^19] |
| PaLM 2 | SentencePiece | larger than PaLM | Tokenizer expanded for multilingual coverage[^20] |
| DeepSeek-LLM, DeepSeek-V2 | SentencePiece BPE | 100,000+ | byte_fallback enabled[^21] |
| Whisper | Adapted from GPT-2 BPE | 50,257 / 51,865 | Custom multilingual tokenizer, not SentencePiece[^22] |

### T5, mT5, ALBERT, and XLNet

[T5](/wiki/t5) introduced the text-to-text framing of NLP tasks and used SentencePiece with a unigram model and a 32,000 piece vocabulary trained on a filtered subset of Common Crawl; the choice of unigram over BPE was motivated by the smoother vocabulary statistics and the natural support for subword regularization.[^5] [mT5](/wiki/mt5), the multilingual extension covering 101 languages, expanded the SentencePiece unigram vocabulary to 250,000 pieces to accommodate the wider Unicode coverage required for global text.[^6] [ALBERT](/wiki/albert), a 2019 lighter-weight variant of BERT, used SentencePiece with a unigram model and a 30,000 piece vocabulary, in contrast to BERT's WordPiece.[^7] [XLNet](/wiki/xlnet), the 2019 autoregressive Transformer that competed with BERT on GLUE-style benchmarks, also adopted SentencePiece with a 32,000 piece vocabulary.[^8]

### PaLM and PaLM 2

Google's PaLM, a 540 billion parameter dense decoder-only model introduced in 2022, used SentencePiece with a 256,000 piece vocabulary that included extensive coverage of code tokens and multilingual text.[^19] PaLM 2, the follow-up that appeared in 2023, used a further expanded SentencePiece vocabulary to support stronger multilingual reasoning.[^20]

### Llama 1, Llama 2, and the transition to tiktoken in Llama 3

Meta's first [Llama](/wiki/llama) release in February 2023 used a SentencePiece BPE tokenizer with a 32,000 piece vocabulary and `byte_fallback=True`, trained on a mostly English-centric corpus.[^10] [Llama 2](/wiki/llama_2), released in July 2023, used the same tokenizer family, retaining the SentencePiece BPE setup with a 32,000 piece vocabulary and byte fallback.[^17]

[Llama 3](/wiki/llama_3), announced in April 2024, departed from this lineage by switching to a [tiktoken](/wiki/tiktoken)-style byte-level BPE tokenizer with a 128,000 piece vocabulary.[^16][^23] The new tokenizer is implemented through OpenAI's tiktoken library rather than SentencePiece, and the larger vocabulary results in roughly fifteen percent fewer tokens per text for English on average and disproportionately fewer tokens for code and non-English languages.[^23] The shift reflects a broader pattern across frontier English-centric models toward larger, byte-level BPE vocabularies for speed and uniform byte handling.

### Mistral, Gemma, and DeepSeek

Mistral AI's first widely released model, [Mistral 7B](/wiki/mistral_7b) in 2023, used a SentencePiece BPE tokenizer with a 32,000 piece vocabulary, byte fallback enabled, and a setup very similar to Llama 1's.[^18] Later Mistral models such as Mixtral retained the SentencePiece BPE family. Google DeepMind's [Gemma](/wiki/gemma) family of open-weights models, released in 2024, uses a SentencePiece tokenizer with a 256,000 piece vocabulary that is a subset of the tokenizer used by Gemini, with byte fallback enabled.[^11] [DeepSeek](/wiki/deepseek) has used SentencePiece BPE tokenizers across its open-weights releases including DeepSeek-LLM and the DeepSeek-V2 series, with vocabularies on the order of 100,000 pieces and byte fallback enabled.[^21]

### Whisper and clarifications

OpenAI's Whisper speech-to-text model, released in 2022, is sometimes mistakenly listed alongside SentencePiece users but actually uses a tokenizer adapted from the GPT-2 byte-level BPE vocabulary; the multilingual Whisper variant extends the vocabulary to 51,865 pieces.[^22] [BERT](/wiki/bert) likewise does not use SentencePiece; it uses a [WordPiece](/wiki/wordpiece) tokenizer, although WordPiece and SentencePiece's unigram method share conceptual ancestry in likelihood-based vocabulary selection.[^13][^14] Many [GPT](/wiki/gpt) family models from [OpenAI](/wiki/openai) use a custom byte-level BPE that predates and is independent of SentencePiece.

## How does SentencePiece differ from tiktoken?

[tiktoken](/wiki/tiktoken), released as open source by OpenAI in late 2022, is the byte-level BPE tokenizer used by GPT-2, GPT-3, GPT-3.5, GPT-4, and later OpenAI models.[^16] The two libraries solve overlapping problems but make different tradeoffs.

tiktoken operates on raw UTF-8 bytes rather than Unicode code points. Every input is first split into bytes, and BPE merges are learned over byte sequences. This avoids any need for Unicode normalization and guarantees that any byte string can be encoded, no matter how unusual.[^16] SentencePiece, by contrast, operates on normalized Unicode text with NFKC normalization by default and uses the `byte_fallback` option to handle any character outside the trained vocabulary as raw UTF-8 byte tokens. Both approaches reliably encode arbitrary input, but tiktoken's byte-first design handles unusual byte sequences more cleanly without configuration.

tiktoken is implemented in Rust with a thin Python wrapper and is widely benchmarked as several times faster than SentencePiece on encoding workloads. It does not implement unigram language models or expose a training entry point in its public release; users who want to train a new tokenizer on their own corpus must use a different library.[^16] SentencePiece is a full training and inference library and is the standard choice when a model team needs to build a new tokenizer.

In practice, the choice now divides along ecosystem lines. SentencePiece remains the default for open-weights multilingual models, Google's models, and the bulk of academic releases. tiktoken-style byte-level BPE has become the default for OpenAI's GPT family, for Llama 3 and later, and for several other frontier English-centric models where raw encoding speed and byte-level cleanliness are prioritized.[^16][^23]

## How does SentencePiece relate to Hugging Face Tokenizers?

The Hugging Face Tokenizers library, often imported as `tokenizers`, is a Rust-based library with Python bindings that implements BPE, WordPiece, and unigram models in a single modular pipeline.[^15] Its design generalizes SentencePiece's architecture by exposing the normalizer, pre-tokenizer, model, and post-processor as separate stages that can be combined in arbitrary configurations.

Hugging Face Tokenizers can load most SentencePiece `.model` files directly, and offers a fast Rust-based encoding path that is several times faster than SentencePiece.[^15] In the Hugging Face Transformers ecosystem, every SentencePiece-based tokenizer ships with a corresponding "fast" version backed by Tokenizers that supports multi-threaded batch encoding. This is typically the preferred inference path for production workloads.

The most important practical distinction is that Tokenizers is a library of tokenization algorithms with a unified interface, whereas SentencePiece is a specific tokenization library with its own algorithms and file format. Many model teams train their tokenizer with SentencePiece, then convert the resulting model file into the Tokenizers fast format for production serving, gaining both the trained quality and the Rust-based serving speed.[^15]

## What are the limitations of SentencePiece?

SentencePiece has several practical limits, most of which stem from its design vintage rather than algorithmic flaws.

Very large vocabularies, beyond roughly 500,000 pieces, become slow and memory-hungry to train.[^3] Both BPE and unigram training keep candidate counts in memory, so RAM scales with vocabulary size. Unigram training is computationally more expensive than BPE training, by roughly a factor of two to four for the same target vocabulary size, depending on the corpus.[^2][^3]

Detokenization assumes the `▁` whitespace convention. If a downstream system mixes tokens from a SentencePiece tokenizer with tokens from a different tokenizer that uses a different whitespace convention, detokenization can produce unexpected output, which becomes relevant in tool-augmented systems that splice token streams from multiple sources.

Modern frontier large language models are increasingly moving to byte-level BPE implemented in `tiktoken` or in the Hugging Face Tokenizers library, primarily for speed and for cleaner handling of arbitrary input bytes including emoji sequences and control characters.[^16][^23] SentencePiece's unigram model in particular is rare in the latest generation of public frontier model releases, although it remains the dominant choice for multilingual systems.

Serialized model files are tied to the SentencePiece library version's protobuf schema. Forward and backward compatibility have been good in practice, but very old models can occasionally need a migration step when loaded with newer versions, and newer options such as byte fallback cannot be loaded by older library versions.[^3] Finally, the library does not directly support tokenizers that are jointly trained with the downstream model. Research directions that build on or supersede SentencePiece include character-level and byte-level models such as ByT5, Charformer, and CANINE, which dispense with subword tokenization entirely at the cost of longer sequences and additional model computation.

## Is SentencePiece open source?

SentencePiece is distributed under the [Apache 2.0 license](/wiki/apache_2_0_license), which permits commercial use, modification, and redistribution provided that copyright notices are preserved.[^3] The library has been continuously maintained since its 2018 release, with regular releases on PyPI and on GitHub, and is downloaded more than 36 million times per month from PyPI alone, reflecting its position as a default dependency in the open NLP and open-weights model ecosystem.[^3][^24]

Hugging Face Transformers ships SentencePiece-backed tokenizer classes for every model originally trained with SentencePiece, including `T5Tokenizer`, `AlbertTokenizer`, `XLNetTokenizer`, `MarianTokenizer`, `MBartTokenizer`, `NllbTokenizer`, and the `LlamaTokenizer` and `GemmaTokenizer` families.[^15] Many have a corresponding fast version backed by the Hugging Face Tokenizers library that can load the same SentencePiece model file but run encoding in Rust.

## See also

- [Byte-pair encoding](/wiki/byte_pair_encoding)
- [WordPiece](/wiki/wordpiece)
- [Tokenization](/wiki/tokenization)
- [Subword tokenization](/wiki/subword_tokenization)
- [Unigram language model](/wiki/unigram_lm)
- [Subword regularization](/wiki/subword_regularization)
- [tiktoken](/wiki/tiktoken)
- [Hugging Face](/wiki/hugging_face)
- [Taku Kudo](/wiki/taku_kudo)
- [Llama](/wiki/llama)
- [Llama 2](/wiki/llama_2)
- [Llama 3](/wiki/llama_3)
- [Gemma](/wiki/gemma)
- [Mistral 7B](/wiki/mistral_7b)
- [T5](/wiki/t5)
- [mT5](/wiki/mt5)
- [Natural language processing](/wiki/natural_language_processing)
- [Language model](/wiki/language_model)

## References

[^1]: Kudo, T., & Richardson, J. (2018). SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. *Proceedings of EMNLP 2018: System Demonstrations*. https://arxiv.org/abs/1808.06226

[^2]: Kudo, T. (2018). Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. *Proceedings of ACL 2018*. https://arxiv.org/abs/1804.10959

[^3]: Google. SentencePiece GitHub repository. https://github.com/google/sentencepiece

[^4]: Sennrich, R., Haddow, B., & Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. *Proceedings of ACL 2016*. https://arxiv.org/abs/1508.07909

[^5]: Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. *Journal of Machine Learning Research*, 21(140), 1-67. https://arxiv.org/abs/1910.10683

[^6]: Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2021). mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer. *Proceedings of NAACL 2021*. https://arxiv.org/abs/2010.11934

[^7]: Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2020). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. *Proceedings of ICLR 2020*. https://arxiv.org/abs/1909.11942

[^8]: Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R. R., & Le, Q. V. (2019). XLNet: Generalized Autoregressive Pretraining for Language Understanding. *Proceedings of NeurIPS 2019*. https://arxiv.org/abs/1906.08237

[^9]: NLLB Team, Costa-jussa, M. R., Cross, J., Celebi, O., Elbayad, M., Heafield, K., Heffernan, K., et al. (2022). No Language Left Behind: Scaling Human-Centered Machine Translation. *Meta AI technical report*. https://arxiv.org/abs/2207.04672

[^10]: Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., et al. (2023). LLaMA: Open and Efficient Foundation Language Models. *Meta AI*. https://arxiv.org/abs/2302.13971

[^11]: Gemma Team, Google DeepMind. (2024). Gemma: Open Models Based on Gemini Research and Technology. *Technical report*. https://arxiv.org/abs/2403.08295

[^12]: Provilkov, I., Emelianenko, D., & Voita, E. (2020). BPE-Dropout: Simple and Effective Subword Regularization. *Proceedings of ACL 2020*. https://arxiv.org/abs/1910.13267

[^13]: Schuster, M., & Nakajima, K. (2012). Japanese and Korean voice search. *Proceedings of ICASSP 2012*. https://research.google/pubs/japanese-and-korean-voice-search/

[^14]: Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. *Proceedings of NAACL 2019*. https://arxiv.org/abs/1810.04805

[^15]: Hugging Face. Tokenizers library. https://github.com/huggingface/tokenizers

[^16]: OpenAI. tiktoken library. https://github.com/openai/tiktoken

[^17]: Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. *Meta AI*. https://arxiv.org/abs/2307.09288

[^18]: Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., et al. (2023). Mistral 7B. *Mistral AI*. https://arxiv.org/abs/2310.06825

[^19]: Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., et al. (2022). PaLM: Scaling Language Modeling with Pathways. *Google Research*. https://arxiv.org/abs/2204.02311

[^20]: Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., et al. (2023). PaLM 2 Technical Report. *Google*. https://arxiv.org/abs/2305.10403

[^21]: DeepSeek-AI. (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. https://arxiv.org/abs/2405.04434

[^22]: Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2023). Robust Speech Recognition via Large-Scale Weak Supervision. *OpenAI*. https://arxiv.org/abs/2212.04356

[^23]: Meta AI. (2024). The Llama 3 Herd of Models. *Meta AI technical report*. https://arxiv.org/abs/2407.21783

[^24]: PyPI Download Stats. sentencepiece. https://pypistats.org/packages/sentencepiece