Text so far
Candidates for the next token
Settings
Temperature
Top-p
What you are looking at: real output from two real tokenizers. GPT-2's has a vocabulary of 50,257 tokens; GPT-4o's (o200k_base) has 200,019. Each box is one token, with its id underneath. A dot stands for a space, which belongs to the start of the next token. Dashed boxes are pieces of a character: the tokenizer had no merge for it, so it fell back to raw UTF-8 bytes.
Try this: pick Hindi or Tamil and switch between the two vocabularies. Watch the token count drop several times over with the bigger vocabulary, then compare it with English in the bars below. Then pick Numbers and notice how "3.35" is cut.
Bars show tokens per sentence. The Fact sentence is the same meaning in English, French, Hindi and Tamil, so those rows compare languages directly.
Token ids and splits were produced with OpenAI's tiktoken library (gpt2 and o200k_base). Other models have their own vocabularies (Llama 3 uses 128,256 tokens), so their splits differ.
What you are looking at: byte pair encoding, the way most vocabularies are built. Every word starts as single characters, with _ marking the end. Each merge counts every neighbouring pair across the corpus, joins the most frequent pair into one new symbol and adds it to the vocabulary. Real tokenizers do the same on a huge text sample until they reach 50,000 to 250,000 symbols.
Try this: on the Classic corpus press Next merge eight times and watch "newest" become one token. Then type a word it never saw, such as lowest or wider, in the test box: it is built from the learned pieces. Switch to your own words and see which pairs win.
Computed live in your browser. Ties go to the pair seen first. Production tokenizers start from bytes rather than characters and pre-split text on spaces and punctuation, but the merge loop is the same.
What you are looking at: for every step, the model gives every token in its vocabulary a score. The scores become probabilities, and one token is picked at random according to them. Temperature sharpens or flattens the odds; top-p throws away unlikely options. Press Pick next token to write, one token at a time.
Try this: set temperature to 0 and pick through a whole run: the same words every time. Then raise it to 2.0 with top-p at 1.00 and pick on "Capital of France" until " Lyon" or " London" comes up. The wrong answer arrives in the same confident sentence.
The candidate scores are made up for teaching, for a handful of steps per prompt. A real model scores its whole vocabulary at every step using billions of weights. Steps are scripted along the most likely path; picking anything else ends the run. Picking " Lyon" after "The capital of France is" shows how a model can state something false with confidence.
What you are looking at: the generation loop, one step at a time. Step 0 is prefill: every prompt token goes through every layer in one pass and its keys and values are stored in the KV cache. Every later step is decode: only the newest token goes through the layers, reading all the weights and the whole cache, and one token comes out. The numbers are ideal ceilings for one request on H100 SXM GPUs.
Try this: press Next step a few times and compare the prefill step with a decode step. Then add 32,000 tokens of earlier conversation and switch to Llama 3.1 70B: the cache per token grows from 128 KB to 320 KB, and the cache read starts to matter next to the weights.
KV per token uses Llama 3.1's published shapes in 16-bit (8B: 128 KB, 70B: 320 KB). Times divide bytes read by memory bandwidth (3.35 TB/s per H100; the 70B model is spread over two, 6.7 TB/s) and compare with 2 × parameters × tokens at 989 dense BF16 TFLOPS per GPU; real engines reach a fraction of both. Candidate scores are illustrative.
What you are looking at: the same next-token scores under three settings side by side: greedy (temperature 0), a typical setting for this kind of task, and your own. Temperature divides the scores before softmax, top-k keeps the k best tokens, and top-p keeps the smallest set of top tokens that adds up to p. Crossed-out tokens can never be picked.
Try this: on Factual answer, press Sample 200 times with your temperature at 1.5 and top-p at 1.00, and count the wrong cities. Set top-p to 0.90 and sample again. Then switch to Creative writing: the scores are much closer, so the same settings keep far more choices.
Scores are illustrative. Typical settings are common starting points; model makers often publish their own recommended values, which should come first. Counts are random draws, so each press gives slightly different numbers.