Computer
Model
Size (billions of parameters)
Precision (bytes per parameter)
To write each word of an answer, an AI model reads all of its weights from memory. If the whole model fits in the GPU's own memory (VRAM), that read is fast. If it does not, part of the model has to stay in system RAM, and that part is read many times more slowly. Pick a computer and a model to see where it lands.
Speeds are the upper limit set by memory bandwidth for one user: bytes of weights read per word ÷ bandwidth. Real software usually reaches 60–80% of this. Memory needed = weights + about 8% for the KV cache and working memory + 1 GB for the GPU runtime, for a short chat. Sizes are dense models, where every word reads every weight; Mixture of Experts models read only part of theirs (part 12). Part 7 of the series covers sizing in detail.
Every word an 8B model writes in 16-bit precision means reading 16 GB of weights. This race shows how long different kinds of memory take to read those 16 GB once, slowed down so you can watch it. The faster the read, the more words per second the model can write.
Figures are each product's published peak bandwidth. Desktop DDR5 = DDR5-5600, two channels. Server DDR5 = DDR5-4800, twelve channels on one CPU socket. The PCIe and NVMe lanes show why loading a model, or reading part of it across the PCIe link, is slow.