What you are looking at: one chat request going through a serving engine. Text comes in over HTTP, is laid out with the chat template, turned into token ids, waits in the queue, is read in one prefill pass, then the answer is produced one token per decode step and streamed back as server-sent events. The last event carries the token count used for billing.
Try this: pick "Coding agent" and press Send. Most of the wait before the first word is prefill of the long prompt. Then raise "Requests already waiting" and notice that the first word now waits on the queue, while the speed of the words once they start does not change.
What you are looking at: the files in the Llama 3.1 8B Instruct repository, with their real sizes. Almost the whole folder is weights. The small files decide how the engine builds the network, how text becomes tokens and how chat messages are laid out. Click a file to see what it holds and what reads it.
Try this: change the precision to FP8 and the GPU to an L40S, and watch how much memory is left for the KV cache, and how many tokens of conversation that holds.
What you are looking at: how long a brand-new model server takes before it can answer, split into its stages. The weights have to be fetched, copied into GPU memory, and the engine has to measure memory, reserve the KV cache and prepare GPU programs before it reports ready.
Try this: pick 70B and switch between local NVMe and internet download. Then raise the traffic that arrives during start-up and see how many requests pile up before the new server can help.
What you are looking at: one server under steady traffic. The engine runs many requests together in one batch. More requests per second means a bigger batch, so each decode step takes a little longer, and prefill of new prompts takes a bigger share of the GPU. Near the limit, requests start to wait in the queue and time to first token climbs steeply.
Try this: with "Chatbot" selected, move the traffic slider slowly to the right and watch where the red line bends. Then switch to "RAG over documents": the long prompts fill the KV cache with far fewer requests, so the same server bends at a much lower rate.
All timings are illustrative, for 16-bit models on H100s (8B on one GPU: about 20,000 prompt tokens per second and 8 ms per output token when alone; 70B on four GPUs with tensor parallelism: about 8,000 prompt tokens per second and 20 ms per token). The load view is a simple model: each decode step gets slightly slower for every request in the batch, prefill takes its share of GPU time, the batch is capped at 256 requests or by KV cache space (about 400,000 tokens in both set-ups), and queueing uses the Sakasegawa approximation. Cold start uses sustained read speeds of 3 GB/s for local NVMe, 1 GB/s for network storage and 100 MB/s for an internet download, a 10 GB container image pulled at 500 MB/s, about 25 GB/s per GPU to load over PCIe, and 40 seconds to profile memory and capture GPU graphs. File sizes are from the Llama 3.1 8B Instruct repository. Real numbers depend on the engine version, settings, storage and network.