Tokenization (Deep)
You know tokens split text into pieces. Now see how the vocabulary is actually built — Byte-Pair Encoding merges the most common character pairs 50,000 times until common words earn their own token. And why token count is the unit that drives every cost, limit, and performance number in AI.
An odd cut
The AI cuts “unhappiness” into “unh” and “appiness”. Not “un” and “happiness”.
No person would cut it there. So who decided where the cuts go?
Why not letters?
Cutting by letter would work, but this word would take 11 pieces instead of 2.
Longer inputs mean more work at every step, and one letter on its own carries almost no meaning.
Count and glue
Before any training, a program counted which pairs sit side by side most often, and glued them together.
Glue the top pair, count again, glue again. After many thousands of rounds, common words are one piece.
Common or rare
Tap a word. Common words are one piece; rare ones are built from parts.
Each word is cut the way it would be in the middle of a sentence.
Numbers too
Numbers get cut too. Tap one to see how.
Each number is cut as if typed on its own.
Sort the words
One piece or several? Tap where each word belongs.
Think about how often you’ve seen the word written down.
Other languages
The same question takes 5 pieces in English and 9 in Japanese.
This tokenizer gave English the most whole-word pieces. Other languages often need more for the same meaning.
Counted in pieces
Five words, nine pieces. And pieces are what everything is measured in.
How much text fits, how fast it replies and what it costs are all counted in pieces, not words.
Found by counting
The pieces were found by counting and gluing common pairs. Everything is measured in them.
The gluing method is called byte-pair encoding. The full set of pieces is the vocabulary.