Revision History
Tokenization · 13 revisions
Sizes are character counts of the article source. The signed number is the change from the previous revision.
Recent edit summaries
Detailed summaries recorded by editors. Generic maintenance summaries are omitted here; every recorded revision remains below. Dates describe the edit, not necessarily the event it covers.
- Correction: Corrected GPT-4 (cl100k_base) tokenization examples and the false claim that 3-digit numbers like 481 split irregularly, fixed the cl100k regex digit rule, the GPT-5/o200k_harmony and Gemma 3 Unigram/BPE errors, replaced unsupported Cyrillic-token figures, and fixed the misattribution of the 'distribute' glitch-token response to ChatGPT (plus 28 minor fixes)
Version 13 · Nov 1, 2026, 11:32 AM
- Update: added a video.
Version 12 · Nov 1, 2026, 08:11 AM
- Improve internal links (2026-10-13)
Version 11 · Oct 13, 2026, 10:07 AM
- Added 6 contextual internal links
Version 10 · Jul 23, 2026, 03:03 PM