Query of " France"
q₁
q₂
What you are looking at: you type "What is the capital of France?" into a Llama-style model. Step through what happens between pressing Enter and the first word of the answer. The shape line under each step shows the size of the data at that point for the model you pick.
Try this: step to the end with the 8B, then switch to 70B and step again. Notice that only the widths and the layer count change; the steps are identical.
What you are looking at: attention lets each word gather meaning from the words before it. Click a word to see where it "looks". The classic example: what does "it" refer to?
Try this: click "it", then "tired", then "street". Each word spreads its attention differently, and words to its right are always grey because a token can never look ahead.
Attention weights here are illustrative, chosen to show typical patterns. A real model has 32 or more attention heads in every layer, each learning its own patterns, and each word can only look at words before it.
What you are looking at: the worked example from the post. The token " France" has a query of two numbers. It is compared with the keys of " capital", " of" and itself; the scores are scaled, turned into weights by softmax, and used to mix the three value vectors.
Try this: move the query towards [2, 0] and watch the weight move from " capital" to " France" itself. Then untick the scaling and see the weights get sharper.
Keys and values are made-up numbers (illustrative). A real head does the same with 128 numbers per vector, for every token, in every head and layer.
What you are looking at: every step inside one transformer layer, with the weight matrix, the shape of the output, the work done (FLOPs) and the bytes read from GPU memory, for a prefill of your prompt or one decode step. Click a row or press Next to walk through it.
Try this: pick Decode with 1,000 tokens of history, then drag the history to 128K. The weight reads stay the same, but the attention rows grow until the KV cache read is larger than the weights. Then switch to Prefill and see the bottleneck move from memory to compute.
16-bit weights and KV cache; KV cache sizes in binary units (1 KB = 1,024 bytes), weights in decimal GB. Times are ideal H100 SXM ceilings (989 TFLOPS BF16, 3.35 TB/s), illustrative: real kernels reach a fraction of both. Element-wise steps (norms, RoPE, softmax, SiLU, adds) are shown with no FLOPs because they are tiny next to the matrix multiplications.
What you are looking at: where the parameters of Llama 3 8B and 70B sit, counted from the published shapes, and where the work for one new token goes as the history grows. The last chart shows how many key-value heads the model keeps, and what that does to the KV cache.
Try this: pick the "Coding agent" preset on the 8B: attention work passes the weight work. Then set KV heads to 32 to see the cache the model would need without grouped-query attention.