The Attention Mechanism
Every token produces three vectors — Query, Key, Value. Attention scores every Q·K pair, scales by √d_k, applies softmax, then blends the Value vectors by those weights. Run this 32 times in parallel with different weight matrices and you get multi-head attention — each head learning a different type of relationship.
Who is “it”?
You know words look back at each other. But how does “it” know to look at “cat”?
It can only look back, and it can’t look up meanings. It needs a way to score every earlier word.
Each word asks
Each word makes a query: a description of what it’s looking for.
Like everything else in the model, the query is a list of numbers, made from the word’s embedding.
Each word offers
Each word also has a key: a description of what it has to offer.
A query and a key that point the same way make a high match score.
You be the query
You be the query. Tap the word whose key best answers the highlighted word.
Scores become shares
Every query meets every key. The match scores are turned into shares that add up to 100%.
Strong matches take most of the share; weak ones get almost nothing. That step is called softmax.
Values get blended
Each word also carries a value: what it passes on. “it” blends the values by those shares.
So “it” comes out mostly made of what “cat” passed on. Query, key, value: Q, K and V.
Many heads
This happens many times side by side. Each copy, called a head, learns to look for something different.
Query, key, value
Each word sends a query, matches it against every key, and blends the values it finds.
Turning scores into shares is softmax. Running many of these side by side is multi-head attention.