How Models Are Trained
Training a large language model is an engineering challenge as much as a research one. Fifteen trillion tokens of curated data, 8,192 GPUs running for three months, a loss curve that starts at 6.4 and slowly grinds to 1.4 — and at the end, capabilities emerge that nobody explicitly trained for.
An untaught skill
Nobody wrote a translation rule into this model.
It was only ever trained to guess the next piece of text. So where did the French come from?
The same loop, scaled
Training is the loop from lesson 7: guess, check, nudge. Only the scale changes.
Large models run it over trillions of pieces of text, on thousands of chips, for weeks or months.
Keep or drop
First the text gets cleaned. Would you keep this, or drop it?
Tap a box for each kind of text.
Early guesses
Early in training it reads “The capital of France is”. What does it guess next?
Fast, then slow
Its error falls fast at first, then slowly. Each later gain needs far more text.
Common words come first, then grammar, then facts and longer reasoning.
Step size
Each nudge has a size. Too small and too big both go wrong.
Back to bonjour
Translation was never a goal. It came from guessing well on text that included French.
To predict what people write, a model has to pick up some of what people know.
One loop, at scale
Pre-training runs one simple loop over a huge pile of cleaned text.
The error it shrinks is the loss. The size of each nudge is the learning rate.