tokenizer bpe

Byte-Pair Encoding (BPE) is a subword tokenization algorithm that iteratively merges the most frequent character pairs — producing a vocabulary of subword units that balances vocabulary size with sequence length and handles unknown words gracefully.

Why Tokenization Matters

• LLMs process tokens, not characters or words.
• Word-level vocabulary: 500K+ words, fails on unseen words.
• Character-level: Very long sequences, slow training.
• Subword (BPE): Best of both — compact vocabulary, handles rare words.

BPE Algorithm

1. Initialize vocabulary with individual characters.
2. Count frequency of all adjacent byte/character pairs.
3. Merge the most frequent pair → new token.
4. Repeat until vocabulary size V is reached (typically 32K–100K).

Example:
- "l o w", "l o w e r", "n e w" → merge most frequent "ow" → "low", "lower", "new"
- Result: common words become single tokens; rare words split into subwords.

Tokenizer Variants

• BPE (GPT-2, GPT-3, LLaMA): Operates on bytes, handles any Unicode.
• WordPiece (BERT): Like BPE but maximizes likelihood of training data instead of frequency.
• SentencePiece (LLaMA, T5): Language-independent, treats whitespace as a token.
• Unigram (ALBERT): Probabilistic subword model — prunes tokens that minimize overall likelihood.

Tokenization Impact on Models

• Number of tokens per word varies by language — English ~1.3 tokens/word, Chinese ~2-3 tokens/word.
• Code tokenizers often use code-specific BPE (dedented whitespace, common identifiers).
• Tokenization artifacts can cause reasoning errors (e.g., counting letters in words).

Vocabulary Sizes

Tokenization is a foundational but often overlooked design decision — vocabulary size, granularity, and algorithm directly affect training efficiency, multilingual performance, and arithmetic reasoning.