A Gemma 27B fine-tune I moved onto a single v6e-8 slice fell over with an out-of-memory error the moment I raised the sequence length. The chips were not slow. There simply were not enough of them wired together on one slice to hold the optimizer states. That is the wall Part 18 never hit, because serving one model on eight chips is a different problem from training a large one across hundreds. This part is about crossing that wall: how a slice becomes a Pod, how Pods join into a Multislice job over the data center network, and how Google Kubernetes Engine actually schedules the whole thing.
One slice runs out of memory
Start with the numbers. Each v6e chip carries 32 GB of high bandwidth memory, or HBM, the fast on-package memory a TensorCore reads and writes during a step. A TensorCore is the matrix engine inside the chip. Put eight chips on one host, the largest single-host v6e slice, and you have 256 GB of HBM to work with. That sounds generous until you write out where training memory actually goes.
A model in training holds far more than its weights. For a model with P parameters in bfloat16 you spend 2P bytes on weights and another 2P on gradients. The Adam optimizer then keeps a full-precision copy of the weights plus two moment estimates, which is roughly 12P bytes more. Backward-pass activations sit on top of that. The weights are the small part. The optimizer is what sinks you.
So a single slice is not slow, it is small. A slice is a set of TPU chips wired together with inter-chip interconnect, the on-fabric links that let chips trade tensors without touching the host network. A Pod is the largest block of chips joined by that same fabric. On v6e a Pod is 256 chips arranged as a 2D torus, a grid whose edges wrap so every chip has near neighbours on all sides. Past 256 chips the ICI fabric ends. That boundary is the entire reason Multislice exists.
Slices, Pods, and the two networks between them
Two networks decide how a large job behaves, and they are not close in speed. Inside a slice, chips talk over ICI. On v6e each chip has four ICI ports and 800 GBps of bidirectional bandwidth, and those links never leave the slice. Between slices, traffic rides the Data Center Network, the DCN, which is ordinary high-speed Ethernet between hosts. Each v6e host carries four 200 Gbps network cards, so a full 256-chip Pod has 25.6 Tbps of DCN bandwidth. Divide that across 256 chips and a single chip sees roughly 12 GBps of cross-slice bandwidth, against 800 GBps of ICI on its own fabric.
That gap is not a detail. It sets the entire design rule for the job: keep the chatty, per-step collectives on ICI and let only the infrequent, batched syncs cross the DCN. The chart puts the three memory and network paths a chip sees on one axis so the ratio is hard to forget.
| Path | Scope | v6e bandwidth | What you put on it |
|---|---|---|---|
| ICI | Within a slice or Pod | 800 GBps per chip, 4 ports | Per-step FSDP all-gather and reduce-scatter |
| DCN | Between slices | 25.6 Tbps per 256-chip Pod | Cross-slice data-parallel gradient sync |
| HBM | On each chip | 1638 GBps, 32 GB capacity | Weights, gradients, optimizer, activations |
Where does the model actually get split?
Two forms of parallelism carry almost every large TPU job, and each maps to one of those networks. The first is fully sharded data parallelism, or FSDP. Rather than replicate the full model on every chip, FSDP splits the weights, gradients, and optimizer states across the chips in a slice. Before a layer runs, the chips all-gather the shard they need over ICI; after the backward pass they reduce-scatter the gradients the same way. Because that traffic fires on every step, it has to live on the fast fabric. FSDP over ICI is the default inside a slice.
The second is plain data parallelism across slices. Each slice holds one full replica of the sharded model and trains on a different shard of the batch. Once per step the replicas all-reduce their gradients so every slice ends up with the same update. That all-reduce is the only traffic that crosses the DCN, and it happens once per step instead of once per layer, which is exactly why the slower network can absorb it. This is the pattern MaxText, the reference JAX training stack, uses to scale from one Pod to tens of thousands of chips. Training stays synchronous: every slice steps in lockstep, so a straggler slows all of them.
FSDP and data parallelism cover most jobs, but two other splits appear once models get very large. Tensor parallelism cuts individual matrix multiplies across chips, so a single layer runs on several chips at once; it is bandwidth-hungry and belongs on ICI inside a slice. Pipeline parallelism puts different layers on different chips and passes activations down the line, which tolerates slower links and can even span slices over the DCN. MaxText lets you combine these with FSDP through a few config flags rather than rewriting the model. Begin with FSDP over ICI and data parallelism over the DCN, and add tensor or pipeline splits only when a profile shows you need them.
Sizing a slice before you scale out
Before reaching for Multislice, size the largest single slice that clears your memory budget. v6e slices come in fixed 2D shapes, each mapping to a machine type and a host count. The table is the menu you actually pick from.
| Topology | Chips | Hosts | HBM total | Machine type |
|---|---|---|---|---|
| 2×4 | 8 | 1 | 256 GB | ct6e-standard-8t |
| 4×4 | 16 | 2 | 512 GB | ct6e-standard-4t |
| 4×8 | 32 | 4 | 1024 GB | ct6e-standard-4t |
| 8×8 | 64 | 8 | 2048 GB | ct6e-standard-4t |
| 16×16 | 256 | 32 | 8192 GB | ct6e-standard-4t |
Worked example
Take a 70B-parameter model. Weights in bf16 are 70e9 times 2, so 140 GB. Gradients add another 140 GB. Adam keeps an fp32 master copy plus two moment vectors, about 12 bytes per parameter, so 840 GB. That is 1120 GB before a single activation. A 4×8 slice gives 1024 GB of HBM, which does not clear it. The next step up, an 8×8 slice at 2048 GB, holds the model with room for activations and a working batch. So the answer is a single 64-chip slice, not Multislice yet. You reach for a second slice when you want more throughput or a larger global batch, not because the model failed to fit.
How GKE schedules a Multislice job
On GKE a slice is a multi-host TPU node pool: a group of VMs whose chips share one ICI fabric. A Multislice job is several of those node pools running one workload. You describe it with a JobSet, a Kubernetes resource that groups related Jobs, using one replicated job per slice. Kueue, the job queueing controller, holds the JobSet until all the capacity is free, then admits it as a unit so you never get half a job running and burning money.
flowchart LR A[Submit JobSet] --> B[Kueue admits all or nothing] B --> C[GKE scheduler] C --> D[Node pool slice 1] C --> E[Node pool slice 2] D -->|ICI FSDP| F[16 chips one replica] E -->|ICI FSDP| G[16 chips one replica] F -->|DCN all-reduce| G
Run a two-slice job on GKE
Here is the smallest thing worth calling Multislice: two v6e-16 slices, 32 chips in total, running one MaxText job. Each 4×4 slice needs 16 chips divided by 4 chips per VM, so four nodes per slice. Create one node pool per slice against a reservation, with compact placement so the VMs land on the same ICI domain.
# one node pool per slice; repeat with np-slice-2 gcloud container node-pools create np-slice-1 --location=us-east5 --cluster=tpu-cluster --node-locations=us-east5-b --machine-type=ct6e-standard-4t --tpu-topology=4x4 --num-nodes=4 --reservation-affinity=specific --reservation=my-v6e-res --placement-type=COMPACT
The --tpu-topology=4x4 flag is what makes this a multi-host slice; --num-nodes=4 must equal the chip count divided by four. The accelerator label GKE assigns follows the pattern tpu-v6e-slice.
apiVersion: jobset.x-k8s.io/v1alpha2
kind: JobSet
metadata:
name: multislice-v6e
annotations:
alpha.jobset.sigs.k8s.io/exclusive-topology: cloud.google.com/gke-nodepool
spec:
failurePolicy:
maxRestarts: 4
replicatedJobs:
- name: slice
replicas: 2 # two v6e-16 slices joined over the DCN
template:
spec:
parallelism: 4 # 16 chips / 4 per VM = 4 nodes
completions: 4
backoffLimit: 0
template:
spec:
nodeSelector:
cloud.google.com/gke-tpu-accelerator: tpu-v6e-slice
cloud.google.com/gke-tpu-topology: 4x4
containers:
- name: trainer
image: REGION-docker.pkg.dev/PROJECT/REPO/maxtext:latest
resources:
limits:
google.com/tpu: 4
command: [bash, -c, python3 MaxText/train.py MaxText/configs/base.yml run_name=demo]# expected: 8 pods, two slices of four, all Running $ kubectl get pods -l jobset.sigs.k8s.io/jobset-name=multislice-v6e NAME READY STATUS RESTARTS AGE multislice-v6e-slice-0-0 1/1 Running 0 4m multislice-v6e-slice-0-3 1/1 Running 0 4m multislice-v6e-slice-1-3 1/1 Running 0 4m
Failure mode: pods stuck in Pending with FailedScheduling. Because a multi-host node pool scales atomically from zero to full size, one missing host or a reservation that does not cover the topology leaves the whole slice at zero and nothing schedules. Check the reservation first, then quota.
Ironwood changes the node-pool contract
The version baseline shifted this year. TPU7x, the first chip in the Ironwood family and Google’s seventh TPU generation, reached general availability on 22 April 2026 and is positioned inference-first. The node pool command changes with it. For an Ironwood multi-host slice you first create a workload placement policy, unique per project, region, and topology, then pass --placement-policy and --accelerator-topology instead of the --tpu-topology flag you use on v6e. The machine type is tpu7x-standard-4t, four chips per VM as before.
Where Multislice wastes money
Multislice fails quietly more often than it fails loudly, and the failures are expensive because the meter runs the whole time. The DCN all-reduce is the first trap. If each slice is small and you run many slices, the once-per-step gradient sync starts to dominate, and adding slices buys less than the chip count suggests. Keep each slice as large as your model allows so the fast ICI work stays high relative to the slow cross-slice sync.
Gotcha
Spot VMs look tempting on a big slice until you remember multi-host slices are all-or-nothing. One preempted host takes the entire slice down, and the JobSet restarts from the last checkpoint. If you use Spot, checkpoint often with Orbax and turn on autocheckpoint so a preemption saves state before the VM goes. Elastic training in MaxText can drop a slice and keep going, but only if your run was built for it. On reserved capacity none of this bites.
Three more things catch people. Every slice in a Multislice job must share the same TPU type, size, and topology, so you cannot mix a 4×4 and a 4×8. TPU v3 does not support Multislice at all. And the number that tells you whether any of this is working is goodput, the share of wall-clock time spent on useful compute rather than restarts and stalls. Measure it from the start with the goodput library or the tpu-info CLI, because a job that is 60 percent goodput is quietly doubling your bill.
Checkpoint cadence decides your restart cost
Scale changes the math on checkpointing. On one host a crashed run costs you minutes. Across many slices, a failure late in an epoch can throw away hours of work on hundreds of chips, and the cost of a lost step climbs with every chip you add. That is why checkpoint cadence is a scaling decision, not an afterthought. Orbax, the JAX checkpointing library MaxText uses, writes sharded checkpoints in parallel so a save does not stall every chip while one host talks to storage.
Two settings carry most of the weight. Asynchronous checkpointing lets training keep running while the previous checkpoint flushes to Cloud Storage, so a frequent cadence stops eating throughput. Autocheckpoint catches a pending maintenance or preemption signal and writes state before the VM goes, which turns a lost slice into a resume instead of a restart. Tune the cadence against your goodput target: checkpoint often enough that a failure costs less than the gap between saves, and no more often than that. I restore from a checkpoint once before the real run, every time, because a checkpoint you have never loaded is a guess, not a backup.
Single slice until the model stops fitting
My rule is boring on purpose. Stay on one slice as long as the model, its optimizer states, and a working batch fit in one Pod at your target sequence length. A single slice keeps every collective on ICI, needs no reservation gymnastics, and gives you one node pool to reason about instead of a JobSet plus Kueue. Go Multislice for two reasons only: the model needs more than 256 chips of HBM, or you want more throughput than one Pod delivers and can afford the DCN sync. Do not go Multislice for a small model, and never for latency-sensitive serving, where a second slice adds cross-slice hops for nothing.
Before the first multi-slice run, validate three things: a reservation that covers your exact topology, a checkpoint cadence you have restored from at least once, and a goodput number you trust. Get those right and Multislice scales close to linear. Skip them and it scales your invoice instead. Part 20 turns to the data side of training, where the input pipeline and embeddings decide whether all these chips stay fed. If you want to see where distributed training sits in the full platform, the pillar guide maps every part.
References
- Cloud TPU v6e architecture and supported configurations
- Deploy TPU Multislices in GKE
- Cloud TPU Multislice introduction
- TPU7x (Ironwood) documentation
About The Author
DrJha