What you are looking at: one token arriving at one MoE layer with 8 experts. The router multiplies the token's vector by a small matrix and gets one score per expert. Press Next step to turn the scores into probabilities, keep the top experts, rescale their weights and mix the experts' outputs. Every number in the table and the box below is computed from the scores.
Try this: step through with top-2, then switch to top-1 and top-4 and watch the mixing weights change. Then pick the token "1" with its close scores: the weights come out near 50/50, so both experts matter almost equally. Switch to sigmoid scoring (DeepSeek-V3 style) and notice the probabilities no longer add up to 1 until the chosen ones are normalised.
Scores and expert outputs are made-up example numbers; real tokens have thousands of values and real routers are learned. Expert size uses Mixtral 8x7B's shape: each expert has 3 × 4,096 × 14,336 ≈ 176M weights, and the router is 4,096 × 8 weights. Softmax over all experts then renormalising the top-k gives the same weights as Mixtral's softmax over the top-k scores. DeepSeek-V3 scores each expert with a sigmoid and normalises the chosen scores.
What you are looking at: one layer of a real MoE model. Each square is an expert. Click a token to see which experts the router picks for it; the blue square is a shared expert that every token uses. The cards compare the weights that must be stored with the weights this token actually uses.
Try this: click through the tokens on Mixtral, then switch to DeepSeek-V3. Mixtral uses 2 of 8 experts (28% of its weights), DeepSeek-V3 8 of 256 plus one shared (about 6%). Notice that the "total" card never changes when you click a token: memory is set by every expert, not by the ones in use.
Model figures from the makers: Mixtral 8x7B 46.7B total and 12.9B active, 8 experts with 2 per token; gpt-oss-120b 117B total and about 5.1B active, 128 experts with 4 per token; DeepSeek-V3 671B total and 37B active, 256 routed experts with 8 per token plus one shared expert. Expert choice here is random and per layer; real routers learn patterns.
What you are looking at: a serving engine puts many users' tokens into each step. Every token picks its own experts, and the GPU must read the weights of every expert that at least one token picked. The grid shows one layer; dark squares are experts woken by the batch.
Try this: on DeepSeek-V3, slide from 1 to 128 tokens. Weights read per step climb from about 37 GB to about 660 GB, close to the full model, while weights read per token fall. Watch "tokens per woken expert": even at 128 tokens each expert gets only about 4, which is far too few to keep a GPU's maths units busy.
Weights counted at 8-bit (1 byte each). The expected share of experts woken is 1 − (1 − k/N)B, which assumes each token picks experts independently; real tokens are correlated, so real counts are somewhat lower. The always-used part (attention, embeddings, dense layers, shared experts) is estimated from each maker's total and active figures. Illustrative.
What you are looking at: experts spread across GPUs (expert parallelism). Each GPU runs attention for its own tokens, then sends every token copy to the GPU that holds the chosen expert (dispatch), and gets the results back (combine). Press Animate one layer to watch a sample of tokens travel. The numbers show the traffic for a whole forward pass and how long the link needs for it.
Try this: start with DeepSeek-V3 shape on 8 GPUs over NVLink, then switch the link to PCIe and to a 400 Gb/s network: the same traffic takes 7 to 9 times longer. Then tick "One hot expert" and watch the busiest GPU's load climb above the average; every other GPU waits for it. Finally try 8-bit dispatch, which halves the traffic.
Traffic per GPU per direction = tokens per GPU × experts per token × hidden size × bytes × MoE layers × (GPUs − 1) ÷ GPUs. Shapes: Mixtral 8x7B hidden 4,096, 32 MoE layers, 8 experts, top-2; DeepSeek-V3 hidden 7,168, 58 MoE layers, 256 routed experts, top-8. Link speeds per direction: NVLink 4 450 GB/s, PCIe 5.0 x16 about 64 GB/s, 400 Gb/s port 50 GB/s, 100 Gb/s port 12.5 GB/s. Ideal transfer times with no overlap with computing; real engines overlap some of it. Routing is random. Illustrative.
What you are looking at: an MoE model next to a dense model, on the GPU you choose. Memory follows total parameters; maths per token follows active parameters; weights read per step depend on the batch. Bars compare the two models on each measure.
Try this: pick "Top quality" (DeepSeek-V3 vs Llama 3.1 405B) on H200: the MoE model needs more GPUs but does about 11 times less maths per token and is much faster for one user. Now slide the batch to 128 and watch the single-step read of the MoE model approach its full size. Then try "One 80 GB GPU" with 4-bit weights.
Weights = total parameters × bytes per weight (4-bit counted as 0.53 bytes to include block scales). GPUs needed = weights × 1.1 ÷ GPU memory, rounded up, before any KV cache. Maths per token = 2 × active parameters. Speed ceilings are ideal: weights read per step ÷ combined memory bandwidth of the GPUs, ignoring KV cache, communication and overheads, so real speeds are lower. Step time is the larger of the read time and the maths time at dense BF16 (989 TFLOPS per GPU) for 16-bit, or FP8 (1,979 TFLOPS) for 8-bit and 4-bit, at full efficiency. GPU figures: H100 80 GB 3.35 TB/s, H200 141 GB 4.8 TB/s, same tensor throughput. Model totals and active counts from the makers. Illustrative.