tokenmonster

Anthropic was already special-casing case-folding in their tokenizers before this recent change: https://transformer-circuits.pub/2025/attribution-graphs/met... "The tokenizer the model was trained with uses a special “Caps Lock” token" (⇪). Their visualizations for Claude 3.5 Haiku also show the Title Case token (↑). This is similar to what the TokenMonster tokenizer does: https://github.com/alasdairforsythe/tokenmonster

The actual analysis: https://github.com/alasdairforsythe/tokenmonster/blob/main/b... > Summary of Findings: > - Comparable (50256-strict-nocapcode) TokenMonster vocabularies perform better than both GPT-2 Tokenizer and tiktoken p50k_base on all metrics. > - Optimal vocabulary size is 32,000. > - Simpler vocabularies converge faster but do not necessarily produce better results when converged. > - Higher compression (more chr/tok) does not negatively affect model quality alone. > - Vocabularies with multiple words per token have a 5% negative impact on SMLQA (Ground Truth) benchmark, but a 13% better chr/tok compression. > - Capcode takes longer to learn, but once the model has converged, does not appear to affect SMLQA (Ground Truth) or SQuAD (Data Extraction) benchmarks significantly in either direction. > - Validation loss and F1 score