Quantization & Inference
A 70B model in bfloat16 weighs 140GB — too big for most GPUs. Quantization compresses each weight to 4 bits, shrinking the model to 35GB with less than 2% quality loss. Understanding quantization formats (GPTQ, GGUF, AWQ), serving infrastructure (vLLM, TGI), and batching strategies is what separates a prototype from a production system.
Too big to load
A model is a long list of numbers, and every number takes memory.
70 billion weights at two bytes each is 140 GB. That’s far more than a laptop has.
Fewer bits
Store each weight with fewer bits. Fewer bits means fewer possible values, so weights get rounded.
Each weight moves only a little. The question is what all that rounding adds up to.
Pick the bits
Fewer bits, smaller model. Somewhere along the way, quality starts to slip.
Where it runs
Where could each one run? Count two bytes per weight at 16 bits, one at 8, half at 4.
Tap the box where each model fits.
Smaller is faster
Writing each token means reading every weight from memory. Smaller weights read faster.
So a quantized model often answers faster too, not only fits.
Test it anyway
Rounding can hurt some skills more than others. Run your evals before you swap models.
Check the tasks you care about most, not just an average score.
Fewer bits per weight
Store each weight in fewer bits: the model gets smaller and faster, and a little rougher.
Rounding weights to fewer bits is called quantization. Running a trained model to get answers is inference.