Tokenization: BPE, WordPiece, and the LLM era

Summary

A tokenizer converts text to model IDs; preprocessing, vocabulary, merge rules and special-token handling belong to the model contract.

Consider a hypothetical corpus containing map five times, maps three times and cap twice. Use UTF-8 byte base pieces, forbid merges across word boundaries, and resolve ties by a declared fixed ordering. Which pairs merge, and how should the learned tokenizer encode maps, cap and unseen traps?

The main families

BPE (Byte-Pair Encoding)

Count adjacent pairs across the weighted corpus, merge the most frequent eligible pair, apply it, then recount. Training changes the segmentation; encoding later replays the learned priority without retraining on each input.

Initially, a+p occurs $5+3+2=10$ times. After merging it, m+ap occurs eight times; after that merge, map+s occurs three times. All three winners are unique, so the tie rule does not affect this example.

The frozen sequence yields maps → [maps], cap → [c, ap] and traps → [t, r, ap, s]. The last word needs available byte pieces t and r, even though the tiny training corpus never used them.

Byte-level BPE

Starting with all 256 byte values makes supported UTF-8 text representable before optional merges. An individual token’s bytes need not form a complete Unicode character; decode the whole sequence with the declared policy. Character-only vocabularies without fallback do not inherit this coverage guarantee.

Change base units or the deployed vocabulary

1. Encode previously unseen Unicode

New task: preserve supported text without an unknown-token fallback or silent normalization.

Picture and change: U+00E9 has UTF-8 bytes [195, 169] before merges. Its decomposed form, U+0065 followed by U+0301, has different bytes. Whether to normalize is a task requirement, not a universal cleanup step.

Code: use the existing tokenizer implementation, pinned here to tiktoken==0.12.0 and cl100k_base. This real encoding is separate from the toy corpus above.

import tiktoken

encoding = tiktoken.get_encoding("cl100k_base")
composed = "\u00e9"
decomposed = "e\u0301"
composed_ids = encoding.encode(composed, disallowed_special="all")
decomposed_ids = encoding.encode(decomposed, disallowed_special="all")
assert list(composed.encode("utf-8")) == [195, 169]
assert encoding.decode_bytes(composed_ids) == composed.encode("utf-8")
assert encoding.decode_bytes(decomposed_ids) == decomposed.encode("utf-8")
assert composed_ids != decomposed_ids
print(composed_ids, decomposed_ids)

Check: both forms round-trip to their original bytes, but their token IDs differ. Special-token-looking strings need a separate explicit policy; do not grant control-token status to arbitrary user text.

Use when: exact code, identifiers or source text must survive preprocessing.

2. Add domain-specific pieces

New task: reduce fragmentation for domain text without silently changing a deployed model’s symbols.

Picture and change: version the vocabulary, IDs, embedding/output rows, templates and corresponding continued training together. Preserving old IDs can avoid one incompatibility, but does not train new rows.

Check: reusing ID 17 for another piece without updating coupled weights changes the model’s input interpretation. Revalidate quality and context counts on the paired model/tokenizer artifact.

Use when: measured domain fragmentation justifies model adaptation and its costs.

Failure and transfer

Serving with different BOS/EOS IDs or a different role template can break compatibility despite ordinary text looking correct. Compare the final serialized IDs, not screenshots of prompts.

Does fewer tokens per document establish a better model?