AI Infrastructure series · Part 12 of 24

DeepSeek-V3 has 671 billion parameters, yet for each token it writes it uses only 37 billion of them. That is a Mixture of Experts (MoE) model, and many of the largest open models now follow the same design: gpt-oss, Qwen3, Llama 4 and Mixtral among them. This part walks one token through an MoE layer with real numbers, then shows what MoE saves (maths per token), what it does not save (memory), why a busy server reads almost every expert anyway, and what spreading experts across GPUs asks of your network.

What is a Mixture of Experts model?

Part 11 showed that a transformer layer has two halves: attention, where tokens look at each other, and an MLP block, where each token is processed on its own. The MLP is where most parameters live: about 70% of Llama 3 8B. In a normal dense model, every token goes through the whole MLP of every layer, so every weight is used for every token.

An MoE model keeps attention as it is and replaces the single MLP with many smaller MLPs called experts, plus a tiny router in front of them. For each token, in each layer, the router scores all the experts and sends the token to only a few of them. The others do nothing for that token. Each expert is an ordinary MLP; what is new is the choice.

The modern form comes from Google research (Shazeer et al., 2017; GShard, 2020, two experts per token; Switch Transformer, 2021, one per token). Mixtral 8x7B brought it to popular open models.

Why bother? Because a model’s knowledge grows with its total parameters, while its running cost per token grows with the parameters it actually uses. MoE separates those two numbers. You get the capacity of a very large model and pay, in arithmetic, for a much smaller one. The rest of this post is about that trade, and about the costs that do not shrink.

What happens to one token inside an MoE layer, step by step?

Each layer has its own router and its own experts, and the same steps repeat in every MoE layer. Here is one token going through one layer of a Mixtral-style model with 8 experts and top-2 routing.

How does the router decide? A worked example

Say the router’s matrix multiplication gives these 8 raw scores (logits) for our token. The softmax raises e to the power of each score and divides by the total, so a score that is higher by 1 is about 2.7 times more likely. The total of the exponentials here is 19.38.

E3 and E6 win. Their probabilities, 0.421 and 0.256, add up to 0.677, so dividing each by 0.677 gives mixing weights of 0.62 and 0.38. Now the two experts run. If E3 returns a vector starting [0.5, -0.2, 0.9, 0.1] and E6 returns [0.1, 0.6, 0.3, -0.4], the layer’s output starts with:

Models differ in the details. Mixtral applies the softmax to the top two scores only, which gives the same weights as above. DeepSeek-V3 uses a sigmoid score for each expert instead of a softmax, picks the top 8 of 256, and then normalises the 8 chosen scores to sum to 1. gpt-oss-120b picks 4 of 128. The idea is the same in each: score, pick a few, weight, sum.

What is a shared expert?

Some knowledge is needed by nearly every token, such as common grammar. With pure routing, many experts each learn their own copy of it. A shared expert is one that every token always uses, outside the router, so the routed experts can spend their weights on less common patterns. DeepSeek-V3 has 1 shared expert and 256 routed experts in each MoE layer, with 8 routed experts chosen per token, so each token runs 9 expert MLPs per layer. Llama 4 Maverick sends each token to its shared expert and one of 128 routed experts. Mixtral and gpt-oss have no shared expert.

Not every layer has to be MoE either: DeepSeek-V3’s first 3 of 61 layers are dense, and Llama 4 alternates dense and MoE layers. So “active parameters” is a figure to take from the maker, not one to work out from the expert count.

What do total and active parameters each cost?

An MoE model has two sizes. Total parameters count every weight, including all experts. Active parameters count the weights one token actually uses: attention, embeddings, any dense layers, shared experts, and the chosen routed experts. Here are the makers’ figures for well-known MoE models.

Share of all weights that one token uses. The trend in newer models is more, smaller experts and a smaller active share.

Each number drives a different cost, and mixing them up is the most common sizing mistake:

Why less compute but not less memory? A worked comparison

Compare DeepSeek-V3 with the largest dense open model of its time, Llama 3.1 405B. DeepSeek’s paper reports its base model ahead of Llama 3.1 405B’s base on most of the benchmarks it lists, so treat them as the same class. Both in 8-bit weights:

Memory bars scaled to 671 GB, compute bars to 810 billion operations per token.

So MoE buys quality with memory instead of compute, which pushes you towards high-memory GPUs (H200, B200) or more of them.

What happens when many users share the GPU?

Step 3 above assumed one user. A serving engine batches many requests into each step (part 17), every token chooses its own experts, and the GPU must read every expert that at least one token chose.

If each token picks k of N experts, and choices were independent, the share of experts woken by B tokens is 1 − (1 − k/N)^B. For DeepSeek-V3, k/N = 8/256 = 1/32. One token wakes 3%. 32 tokens wake 1 − (31/32)³² = 64%. 128 tokens wake 98%.

DeepSeek-V3, assuming about 16.5B always-used weights (attention, embeddings, dense layers, shared experts) and about 654.5B in routed experts, with independent random routing. Illustrative.

Three things follow. First, at large batch an MoE model reads almost its full size every step, like a dense model of the same total size; the “6% active” headline describes one token, not one step. Second, the weight reads per token then match that dense model (671 ÷ 128 = 5.2 GB at 128 tokens), so the lasting saving is the maths: still 11 times less per token. Third, each woken expert processes only a handful of tokens: about 4 at 128 tokens in the step. A matrix multiply with 4 rows cannot keep tensor cores busy, so expert work stays memory-bound until each expert sees dozens of tokens. That is why DeepSeek serves V3’s decode across 320 GPUs with one expert per GPU: the huge combined batch gives every expert enough tokens to do real work.

How are experts kept evenly busy?

Left alone, a router tends to collapse: a few experts start slightly better, get more tokens, improve faster and win even more. The others barely train. In serving, an overloaded expert’s GPU finishes last and everyone waits.

Early MoE designs gave each expert a fixed capacity, a maximum number of tokens it accepts per batch, and dropped the rest. Capacity is the average share multiplied by a capacity factor. Here is how that works in numbers.

To stop collapse, Switch Transformer and GShard added an auxiliary loss: a small extra penalty during training that grows when the share of tokens sent to each expert, and the router’s average probability for it, are uneven. It works, but it pulls against the main goal of predicting the next token, so a strong penalty can hurt quality. DeepSeek-V3 instead uses auxiliary-loss-free balancing: each expert has a bias added to its score only when picking the top 8, not when computing the mixing weights. After each training step, overloaded experts’ biases drop by 0.001 and underloaded ones rise by 0.001, so busy experts become slightly less likely to be picked, with no penalty in the loss. DeepSeek kept only a tiny sequence-level balance loss and drops no tokens in training or inference.

Balance in training does not guarantee balance in production: a coding assistant sends mostly code tokens and may lean on some experts. Serving engines can measure expert load and copy hot experts while running.

How are experts spread across GPUs?

When a model does not fit on one GPU, it must be split (part 5). Dense models usually use tensor parallelism: every GPU holds a slice of every weight matrix and the GPUs combine partial results with an all-reduce each layer. MoE models can also use expert parallelism: each GPU holds a few whole experts, often with attention run data parallel, each GPU handling its own requests.

Expert parallelism needs two exchanges in every MoE layer. After attention and routing, each GPU sends every token copy to the GPU that holds the chosen expert: the dispatch. After the experts run, the outputs go back to each token’s home GPU: the combine. Both are all-to-all operations, meaning every GPU sends a different amount of data to every other GPU at the same moment.

How much traffic is that? A worked example

Take DeepSeek-V3’s shape on one 8-GPU server, sending 16-bit vectors, with 128 tokens per GPU per step (1,024 tokens in total). These are ideal transfer times with no overlap (illustrative).

A decode step on such a server takes tens of milliseconds, so over NVLink the exchange is a small tax, while over PCIe or a network it can rival the computing itself. This is why MoE models benefit so much from big NVLink domains such as the 72-GPU NVL72 rack (part 6), and why multi-server MoE needs the RDMA fabric described in part 8. DeepSeek designed around the same limit: on its H800 cluster NVLink gave about 160 GB/s against 50 GB/s for InfiniBand, so it limited each token to experts on at most 4 servers and overlapped the exchanges with computing. Sending 8-bit vectors instead of 16-bit halves the traffic.

How do you serve an MoE model on your own GPUs?

Size it like any model (weights, plus KV cache, plus about 10% for the runtime, part 7), using the total parameter count for the weights.

Quantize the experts first. Experts hold most of an MoE model’s weights (over 97% in DeepSeek-V3), and each expert is used by only a fraction of tokens, so they tolerate low precision well. gpt-oss ships with only its expert weights in MXFP4 and keeps attention and the router in 16-bit, which is what lets a 117B model fit on one 80 GB GPU. Part 18 covers the formats.

Choose the parallel layout. vLLM, SGLang and TensorRT-LLM all support MoE models. In vLLM, a model on one GPU needs nothing special. For a large MoE on one 8-GPU server, the vLLM documentation’s example runs attention data-parallel and experts expert-parallel:

# gpt-oss-120b: fits one 80 GB GPU
vllm serve openai/gpt-oss-120b

# DeepSeek-V3 on one 8-GPU server: attention data parallel, experts spread over 8 GPUs
vllm serve deepseek-ai/DeepSeek-V3-0324 \
  --tensor-parallel-size 1 \
  --data-parallel-size 8 \
  --enable-expert-parallel

# add the expert load balancer: tracks expert load and adds copies of hot experts
  --enable-eplb

With these flags the expert-parallel size is tensor-parallel size × data-parallel size, here 8. vLLM can use different all-to-all backends (including DeepEP kernels for multi-server setups), and its load balancer can log how balanced the experts are. Option names change between versions, so check your version’s documentation. On Kubernetes, llm-d publishes “wide expert parallelism” recipes for spreading one MoE model over many servers.

Watch the right numbers. High time per output token even at small batch points at GPU-to-GPU bandwidth. Uneven GPU utilisation across an expert-parallel group usually means hot experts, which redundant copies fix.

What makes training an MoE model hard?

Training stores far more per parameter than inference: about 16 bytes per parameter with mixed-precision Adam (part 15). That applies to total parameters, so DeepSeek-V3’s weights, gradients and optimizer state come to about 671B × 16 = 10.7 TB, spread over the cluster, even though each token’s compute is that of a 37B model. Memory again scales with the total and compute with the active part.

Communication is the second problem. DeepSeek-V3 was trained on 2,048 H800 GPUs, with experts spread 64 ways across 8 servers plus 16-way pipeline and data parallelism. Its team wrote a pipeline schedule (DualPipe) that hides the all-to-all behind computing, and set aside 20 of each GPU’s 132 SMs just for communication. Router stability is the third, which the balancing methods above address. Fine-tuning with a small adapter is easier (part 19), but the full base model must still sit in GPU memory.

When is MoE the right choice for self-hosting?

A useful rule of thumb: for a fixed amount of GPU memory, a dense model often gives better quality, because every byte of memory works on every token. For a fixed amount of compute, an MoE model gives better quality, because each token gets the knowledge of a much larger model. Decide which of the two is your real limit, then pick the model type. Part 20 turns this into GPU counts.

What do people get wrong about MoE?

• “Each expert is a specialist in a topic.” Experts are learned, not assigned subjects. The Mixtral paper’s routing analysis found no clear pattern of experts by topic; choices followed syntax and token patterns more than subject matter.
• “A 37B-active model needs the memory of a 37B model.” It needs the memory of its total, 671B. Active parameters do not reduce weight memory at all.
• “MoE is always faster.” It does far less maths per token, but at large batch it reads nearly all its weights per step, and it adds all-to-all traffic. On a weak interconnect, a dense model of similar active size may serve faster.
• “Idle experts can live on the CPU.” Any token can pick any expert on the next step, and with many users nearly all are picked every step. Offloading experts to CPU memory works for a single hobby user at low speed, not for a serving system.
• “Expert parallelism works over any network.” The dispatch and combine happen twice per MoE layer on every step. Without NVLink or a fast RDMA fabric, the GPUs spend their time waiting.

Short answers to common questions

Is the router a separate model?

No. It is one small weight matrix inside each MoE layer, trained together with the rest of the model. It adds a tiny fraction of the parameters.

Does a token use the same experts in every layer?

No. Each layer has its own router and its own experts, so a token’s choices change from layer to layer, and the next token makes its own choices again.

Can I run an MoE model on one GPU?

Yes, if its total weights fit. gpt-oss-120b in MXFP4 fits on one 80 GB GPU, and Meta says Llama 4 Scout fits on one H100 in Int4. Large ones such as DeepSeek-V3 need a full high-memory server.

Do I need expert parallelism?

Only when the model spans several GPUs. Tensor parallelism also works for MoE and is simpler on 2 to 4 GPUs with NVLink. Expert parallelism wins when there are many small experts and many GPUs.

Model figures are the makers’ published numbers: Mistral AI (Mixtral 8x7B), OpenAI’s gpt-oss-120b model card and config, Qwen3-235B-A22B model card, Meta’s Llama 4 announcement, and the DeepSeek-V3 technical report (architecture, routing, auxiliary-loss-free balancing, node-limited routing, training cost and serving layout). Routing history from Shazeer et al. 2017, GShard (Lepikhin et al. 2020) and Switch Transformer (Fedus et al. 2021); expert routing analysis from the Mixtral paper. vLLM flags from the vLLM expert parallel deployment documentation. Llama 3.1 figures from Meta’s paper and model card. Router scores, expert outputs, load bars, batch and traffic figures are worked or illustrative examples.