TL;DR · Key Takeaways
- Attention lets every word in a sentence look at every other word and decide which ones matter for its own meaning.
- It replaced the older approach of reading text strictly left to right, one word at a time, which was slow and forgetful over long passages.
- Because attention compares all words at once instead of in sequence, it runs in parallel, which is exactly what GPUs are built to do fast.
- That combination, better handling of context plus parallel speed, is why the 2017 transformer scaled and why nearly every model since is built on it.
Read this sentence and answer one question: “The trophy would not fit in the suitcase because it was too big.” What does “it” refer to, the trophy or the suitcase?
You said the trophy, instantly, without effort. Now swap one word, “too big” becomes “too small,” and suddenly “it” means the suitcase. Nothing about the word “it” changed. What changed is the relationship between “it” and the words around it, and you resolved that relationship automatically. This is the exact problem a language model has to solve millions of times, and the mechanism it uses has a name that has quietly become the most important word in modern AI: attention. Part 7 left us with static word-vectors, where “bank” gets one fixed point regardless of meaning. Attention is how those vectors come alive and start depending on each other.
The bottleneck attention broke
To see why attention was such a leap, you have to know what came before it. The previous generation of language models, built on recurrent networks (RNNs and their better-behaved cousins, LSTMs), read text the way you might read with a finger on the page: one word at a time, left to right, carrying a running summary of everything seen so far. That running summary was the model’s only memory of the past.
This had two painful problems. The first is forgetting. Cram a whole paragraph into one fixed-size summary and the early details get blurry by the end, the way you lose the start of a very long spoken sentence. A pronoun that referred back to something twenty words ago often lost the thread. The second problem was speed, and it turned out to be the dealbreaker. Because each word had to be processed after the one before it, the work could not be split up. You could throw a hundred graphics cards at the problem and it would not help, since word fifty could not be computed until word forty-nine was done. In an era where progress was coming from sheer scale, a design that refused to parallelise was a dead end.
What attention actually does
Attention throws out the finger-on-the-page habit entirely. Instead of passing a summary forward word by word, it lets every word look directly at every other word in the passage, all at the same time, and pull in whatever is relevant to its own meaning.
Picture each word in the sentence quietly asking the same question: “Which of you helps explain me?” When the model processes “it” in our trophy sentence, attention scores how relevant every other word is to “it.” “Trophy” scores high, “suitcase” scores lower, “the” barely registers. The model then builds a new, updated representation of “it” that is mostly made of “trophy,” effectively merging their meanings. Do this for every word at once and you get something the old models could never produce cheaply: a version of each word that already carries its context. The “bank” sitting next to “river” ends up with a different internal vector than the “bank” sitting next to “loan,” without anyone writing a rule about rivers or money.
That is the whole trick, and it is worth saying plainly: attention is a way of mixing words together in proportion to how relevant they are to each other. A real model stacks this operation dozens of times, each layer letting words attend to the context-enriched versions produced by the layer below. Early layers might link pronouns to nouns; deeper ones capture tone, logic, and long-range structure. Stack enough of these and the model develops a startlingly rich sense of how a passage hangs together.
A concrete case makes it click, and it is where attention was actually born: translation. Render the English “the white house” into French and the word order flips to “la maison blanche,” literally “the house white.” A strict left-to-right model has to hold its breath and hope it lines the words up correctly across the reordering. An attention-based translator instead lets each French word it produces look back and point at the English words it depends on: “blanche” attends straight to “white” even though they sit in different positions. Researchers could literally plot these attention weights as a grid and watch the model discover the alignment on its own, no dictionary of rules required. That visible, sensible behaviour was an early sign the idea was onto something real.
Why this one idea changed the field
The 2017 paper that introduced this was titled, with some swagger, “Attention Is All You Need.” The claim was that you could drop the slow sequential machinery completely and build a model out of attention alone. It worked, and the reason it mattered so much comes down to the parallelism. Comparing every word to every other word is a pile of independent multiplications with no forced order, and that is precisely the workload a GPU eats for breakfast. Suddenly the enormous datasets and the powerful hardware that had been sitting underused had a design that could soak them up.
I will give attention an honest piece of credit and one honest caveat. The credit: almost everything you have heard of, GPT, Claude, Gemini, Llama, runs on this mechanism, and the smooth scaling that made each generation better than the last is downstream of it being parallel-friendly. The caveat, which the next part picks up: comparing every word to every other word means the cost grows with the square of the passage length. Double the text and you roughly quadruple the attention work. That single fact is why context windows are expensive and why so much engineering effort goes into making long inputs affordable.
▾ Go Deeper (optional, for technical readers)
Mechanically, each token’s vector is projected into three smaller vectors: a query (Q), a key (K), and a value (V). The intuition is a lookup. A token’s query is what it is searching for; every token’s key is what it advertises about itself; the value is the content it will actually contribute. To decide how much token A should attend to token B, you take the dot product of A’s query with B’s key. A large dot product means they are aligned, so B gets a high score. Those scores across all tokens are scaled, passed through a softmax so they become weights that sum to 1, and then used to take a weighted sum of the value vectors. The result is the new, context-mixed representation of token A. In matrix form this is softmax(QK⃗ / √d)V, and because it is all matrix multiplication, the whole sequence computes in parallel.
Multi-head attention runs several of these query-key-value systems side by side, each with its own learned projections. One head might learn to track subject-verb agreement, another to link pronouns to their referents, another to follow punctuation or tone. Their outputs are concatenated and combined, so a single layer can attend to several kinds of relationship at once rather than being forced to average them into one. The original transformer used eight heads per layer; large models use far more. The scaling factor √d, the softmax, and the multiple heads are the unglamorous details that make the elegant idea actually train stably.
This is Part 8 of a 30-part walk from zero to the infrastructure behind production AI. The full map is on the Generative AI Complete Guide. If “vectors” needs a refresher, Part 7 covers tokens and embeddings; newcomers can start at Part 1.
The Bottom Line
Strip the mystique away and attention is a mixing rule: let every word weigh every other word and absorb the ones that matter. That solved the two things holding language models back, the forgetting that came from squeezing a passage into one running summary, and the slowness that came from reading in strict order. By comparing everything at once, attention made models both better at context and friendly to the parallel hardware that let them grow.
If I had to nominate a single idea as the hinge the whole field turned on, this would be it. The transformer is not a small refinement of what came before; it is a different shape, and we have spent the years since mostly making it bigger and cheaper rather than replacing it. That cost of comparing every word to every other word is the thread we pull next, in the part on training versus inference and where the real bill comes from.
References
- Attention Is All You Need (Vaswani et al., 2017)
- The Illustrated Transformer (Jay Alammar)
- Attention, visualised (3Blue1Brown)
« Part 7: tokens and embeddings | Generative AI Complete Guide | Next: Part 9, training vs inference »
About The Author
