AI Infrastructure series · Part 3 of 24
People say a GPU has “thousands of cores”, but that is only part of the picture. Inside an AI GPU are small processors grouped into units, special cores built only for matrix maths, and five levels of memory from huge and slow to tiny and instant. This post opens up an NVIDIA H100 and follows one AI calculation through it.
Step through one layer of an AI model as it moves from memory to the Tensor Cores and back, then race a server CPU, CUDA cores and Tensor Cores on the same matrix multiply.
What is inside a GPU?
An H100 is one large chip with about 80 billion transistors, surrounded by stacks of memory on the same package. The chip is organised in layers, like a company: one building, many departments, many workers in each.
An H100 package, simplified. The GPU chip holds 132 streaming multiprocessors around a shared L2 cache. HBM memory stacks sit beside the chip on the same package.
Here are the main parts of an H100 SXM, from the biggest to the smallest:
| Part | How many | What it does |
|---|---|---|
| HBM3 memory | 80 GB | Holds the model’s weights and working data. Feeds the chip at 3.35 TB/s. |
| L2 cache | 50 MB | A shared waiting room between HBM and the SMs, so data used twice is fetched once. |
| Streaming multiprocessors (SMs) | 132 | Small processors that each take a share of the work. All run at once. |
| CUDA cores | 128 per SM, 16,896 in total | General-purpose maths units, one number at a time. |
| Tensor Cores | 4 per SM, 528 in total | Units built only for multiplying blocks of numbers. They do most of the AI work. |
| Shared memory and registers | up to 228 KB and 256 KB per SM | Tiny, very fast memory inside each SM, right next to the cores. |
What is a streaming multiprocessor?
The SM is the GPU’s real working unit. Think of it as a small processor of its own: it has its own cores, its own fast memory and its own scheduler that decides what runs next. A big job is split into pieces and the pieces are handed out to all 132 SMs, which work on them at the same time.
Inside an SM, work runs in groups of 32 threads called warps. All 32 threads in a warp run the same instruction at the same moment, each on different numbers. That is the trick that makes GPUs efficient: one instruction, decoded once, drives 32 calculations. It is also why GPUs are bad at branchy, decision-heavy code, which part 1 covers.
What are Tensor Cores, and why do they matter for AI?
CUDA cores multiply one pair of numbers at a time. Tensor Cores multiply two small blocks of numbers and add the result, all in a single step. AI models are almost entirely made of these block multiplications, so Tensor Cores do most of the work.
The difference is large. These are the H100 SXM’s published peak speeds, in trillions of operations per second (TFLOPS):
Two things stand out. Tensor Cores are about 15 times faster than the CUDA cores on the same chip. And they get twice as fast every time the numbers get half as small, from 16-bit to 8-bit. That is one reason AI keeps moving to smaller number formats, which part 18 of this series covers.
How does an AI model turn into GPU work?
An AI model is a stack of layers. Each layer is mostly a few large multiplications: a table of learned weights times the numbers coming in. The program on the CPU turns each of these into a kernel, an instruction for the GPU, and sends it over. Here is what happens to one of them:
The CPU launches a kernel. It tells the GPU “multiply this layer’s weights by this input” and moves on to prepare the next one.
The work is cut into tiles. The GPU splits the multiplication into hundreds of small tiles and spreads them over all 132 SMs.
Weights stream out of HBM. For one layer of an 8B model, about 117 MB of weights flow from HBM, through the L2 cache, towards the SMs.
Each SM loads its tile into shared memory. Numbers placed here can be reused many times without going back to HBM.
Warps pull numbers into registers. The fastest memory on the chip, right beside the cores.
Tensor Cores multiply and add. Each one works through block after block of the tile.
Results are written back. The next kernel starts on the next part of the layer. An 8B model has 32 layers, each needing several kernels, for every word it writes.
Real AI software fuses several of these steps into one kernel where it can, so data stays close to the cores for longer. That kind of tuning is a big part of why one serving engine can be much faster than another on the same GPU.
Why does a GPU have so many levels of memory?
Because fast memory is small and big memory is slow. The levels form a ladder, and the art of GPU programming is to fetch each number from the slow top of the ladder once, then use it many times at the fast bottom.
The same idea runs through all computing: a CPU has caches for the same reason. What is different in a GPU is the scale. Thousands of cores all want data at once, so every level has to be very wide, and running out of fast memory slows everything down.
What does the CPU do while the GPU works?
The CPU is the manager. It turns your text into tokens, decides which requests to run together, launches the kernels and collects the results. It does very little of the maths.
Launching kernels takes time, and an AI model launches hundreds of them for every word. If the CPU is slow to send them, the GPU sits idle between kernels. Serving engines such as vLLM use a CUDA feature called CUDA Graphs to record a whole sequence of kernels once and replay it with one launch, which removes most of that gap.
Where does the GPU actually spend its time?
Here is the surprise. When an H100 serves one user, the Tensor Cores are idle most of the time. Work out one word of an 8B model in 16-bit:
The cores finish their part almost instantly, then wait for the next batch of weights to arrive. That is why serving many users at once is so important: the weights are read once and used for every user in the batch, so the cores have more to do for each byte that arrives. Part 2 showed that memory speed limits AI; the next part explains exactly when it does, and when the maths becomes the limit instead.
- A GPU is built from streaming multiprocessors (132 on an H100), each a small processor of its own.
- Threads run in warps of 32, all doing the same instruction on different numbers.
- Tensor Cores do the block multiplications AI is made of, about 15 times faster than ordinary CUDA cores.
- Memory comes in levels, from 80 GB of HBM down to registers. Good software reuses each number many times near the cores.
- For one user, the GPU spends far more time waiting for weights than calculating with them.
Figures are NVIDIA’s published H100 SXM specifications; Tensor Core speeds are dense peaks without sparsity. Timings are upper-limit estimates for a single user.
About The Author
DrJha