For a hosted Gemini call you never pick the chip. Google already did, and it serves Gemini on its own Tensor Processing Units. The moment you step off the managed path, to train, fine-tune, or self-host an open model on Vertex AI, the choice lands back on your desk. Pick wrong and you burn a week porting code, or you pay for memory you cannot fill.

This part is about that choice. A Tensor Processing Unit, or TPU, is a chip Google designed only for the matrix math inside a neural network. A Graphics Processing Unit, or GPU, is NVIDIA silicon built first for graphics and now the default accelerator for almost all of AI. Both run on Google Cloud, both are wired into Vertex AI, and the honest answer to which one you want is rarely the one a benchmark slide gives you.

What a TPU actually is

A TPU is an application specific chip. Google built it to do one thing fast: multiply large matrices, which is nearly all of the arithmetic in training and running a neural network. The heart of the chip is a Matrix Multiply Unit, or MXU, wired as a systolic array. Picture a grid of small multipliers where numbers flow in from the edges, each cell multiplies and adds, and partial sums march across the grid without a trip back to memory between steps. On Trillium the MXU grew to 256 by 256, which quadrupled the multiply and add operations per cycle over the previous design.

That design has one consequence you feel everywhere. A TPU has very little of the general purpose machinery a CPU or GPU carries, so it is cheap and power efficient per unit of matrix math, and clumsy at anything that is not a big dense tensor operation. Feed it a clean transformer and it flies. Feed it branchy, irregular, sparse code and it stalls. Google chose that trade on purpose, because the models it serves at planet scale are dense transformers, and a chip tuned to exactly that shape costs less to run.

TPUs also do not work alone. They come in slices and pods, groups of chips wired together with a dedicated high speed link called the Inter Chip Interconnect, or ICI, so a large model spreads across many chips as if they were one. Ironwood pushes this to a full pod of 9,216 chips acting as a single training target. That interconnect is a real advantage, because for a model too big to fit one chip, the speed of the wires between chips often matters more than the speed of any single chip.

How a GPU does the same job differently

A GPU reaches the same matrix result by a different road. It packs thousands of small programmable cores running the same instruction on many data elements at once, a model called single instruction, multiple threads, or SIMT. Modern NVIDIA GPUs also carry Tensor Cores, dedicated blocks for matrix math that narrow the raw throughput gap with a TPU. The difference that matters is not peak numbers, it is what the chip will run. A GPU executes arbitrary code, so custom kernels, unusual layers, sparse operations, and the entire research ecosystem run on it without a rewrite.

On Google Cloud the GPUs arrive as accelerator optimized virtual machine families. A3 machines carry NVIDIA H100 chips, and the A3 Ultra variant carries the H200 with 141 GB of memory per GPU. The newer A4 machine packs eight B200 Blackwell GPUs joined by NVLink, and A4X uses the GB200 in an NVL72 rack where 72 Blackwell GPUs act as one large memory pool. That last configuration exists for the same reason TPU pods do: some models are too big for one chip, so the vendor sells you a rack that behaves like one enormous accelerator.

Here is the blunt version. If your model or your training code came from a paper, a Hugging Face repo, or any NVIDIA reference, it already runs on a GPU today with zero porting. That single fact wins more decisions than any throughput chart, and I will come back to it when we talk portability.

The chips you can rent today

Naming is half the confusion, so hold the map in one place. TPUs are versioned by generation. The v5e is the cost efficient member of the fifth generation at 16 GB per chip, tuned for inference and mid size training. The v5p is the heavyweight of that generation at 95 GB per chip, built for the largest training runs. Trillium is the sixth generation, called v6e, at 32 GB per chip, and Google positions it as a large jump in performance for the money over v5e. Ironwood is the seventh generation, at 192 GB of high bandwidth memory per chip, the first TPU Google designed with inference as the primary target rather than an afterthought.

On the GPU side, memory per chip climbs the same way. H100 sits at 80 GB, H200 at 141 GB, and the Blackwell B200 higher still. Memory is the number I look at first, because it sets a hard floor on how many chips a given model needs before anything else enters the conversation. The chart below plots memory per chip across the line, and the spread is the whole story.

Which workload belongs on which chip

Set peak throughput aside and sort by the shape of the work. Large, dense transformer training, where the same clean matrix math repeats for weeks across many chips, is where a TPU earns its keep. Google reports Trillium delivering more than twice the performance per dollar of v5e on dense large language model training, and that gap is the reason its own frontier models train on TPUs. If your job is a long pre training or fine tuning run on a standard architecture and you can write to Google’s stack, the TPU is usually the cheaper finish line.

Serving is more even. A TPU serves high volume, steady traffic on a large model very efficiently, especially with Google’s own inference engine, JetStream, which is the same stack that serves Gemini. But bursty, low volume, or many small models often serve more cheaply on a GPU, because you are not committing to a slice sized for a peak that rarely comes. Research, custom kernels, sparse or unusual architectures, and anything that expects the CUDA ecosystem belong on a GPU, full stop, because the TPU will either refuse the code or run it badly.

Does portability make the decision for you?

Often, yes, and this is the part people underweight. Software decides more accelerator choices than silicon does. TPUs run through a compiler called XLA, which takes your model graph and turns it into TPU instructions. JAX, a Python library Google built for exactly this, is the native path and where TPUs feel best. PyTorch runs on TPUs too, through a bridge called PyTorch/XLA, and TensorFlow works through the same compiler. For serving, JetStream and a TPU build of vLLM, the popular open source inference server, cover most needs.

GPUs run on CUDA, NVIDIA’s software layer, and that ecosystem is the water the whole field swims in. Nearly every model, training script, and kernel published in the last decade assumes a GPU first. The practical result is simple. Code written for TPUs takes real effort to move to a GPU, and code written for GPUs takes real effort to move to a TPU. Whichever you choose, you inherit its stack, and switching later is a project, not a flag.

So the portability question comes down to your starting point. If your code is JAX or already compiles under XLA, a TPU is a natural home and often the cheaper one. If your code is PyTorch with custom operations, third party libraries, and CUDA assumptions baked in, a GPU saves you the port, and the saved engineering time usually dwarfs the hardware price gap. The chip that runs your existing code without a rewrite has a head start that no benchmark overturns.

Sizing an accelerator for a 70B model

Memory per chip is abstract until you put a model against it, so let us serve a real one. Take a 70 billion parameter model in half precision, where each parameter takes 2 bytes. The weights alone need about 140 GB. Serving also needs room for the key value cache that grows with context length and batch size, plus runtime overhead, so plan for roughly 30 percent headroom, about 182 GB. Now the chip count falls straight out of the memory number.

Do not trust my arithmetic, run it. This script has no dependencies and prints the chip count for each accelerator so you can drop in your own model size and memory numbers.

import math

PARAMS       = 70e9    # model parameters
BYTES_PARAM  = 2       # bf16 or fp16 weights, 2 bytes each
KV_HEADROOM  = 1.30    # 30 percent for key value cache and runtime

# usable high bandwidth memory per chip in GB
HBM = {'TPU v5e': 16, 'TPU v6e': 32, 'GPU H100': 80,
       'GPU H200': 141, 'TPU v7 Ironwood': 192}

weights_gb = PARAMS * BYTES_PARAM / 1e9
plan_gb    = weights_gb * KV_HEADROOM
print(f'weights {weights_gb:.0f} GB, plan for {plan_gb:.0f} GB')

for chip, gb in sorted(HBM.items(), key=lambda kv: kv[1]):
    n = math.ceil(plan_gb / gb)
    print(f'{chip:18s} {gb:4d} GB/chip -> {n:2d} chip(s)')

Expected output: weights 140 GB, plan for 182 GB; TPU v5e 16 GB/chip to 12 chips, TPU v6e 32 GB/chip to 6 chips, GPU H100 80 GB/chip to 3 chips, GPU H200 141 GB/chip to 2 chips, TPU v7 Ironwood 192 GB/chip to 1 chip.

Failure mode: this counts weights plus a flat headroom only. It ignores that key value cache grows with context length and batch, that activations need memory during training, and that TPU slices come in fixed topologies, so 6 v6e chips rounds up to a valid 8 chip slice. Treat the number as a floor, then add for real context and parallelism before you provision.

Start on GPUs, commit to TPUs when the model settles

Here is my recommendation without the hedge. For a first training run or a new self hosted model, start on GPUs. Your code almost certainly already runs there, the ecosystem answers every question you will have, and you keep the freedom to change models and libraries while the project is still finding its shape. The hardware may cost a little more per hour, but you spend zero engineering days on a port, and early on that trade favors GPUs every time.

Move to TPUs when three things are true at once: the model architecture is settled and standard, the workload is large and steady enough that price for performance actually moves your bill, and your team can write to JAX or accepts the PyTorch/XLA path. When all three hold, the port pays for itself, and for a dense transformer at real scale a TPU is often the cheaper place to both train and serve. When any one is missing, stay on GPUs and do not apologize for it.

One thing to do this week: take your top model, compute its weight memory the way we did above, and divide by the memory per chip of two accelerators you could actually get quota for. That single number, chips needed, tells you more about your real cost than any performance chart, and it is the first slide I put in front of anyone arguing for a chip on vibes. Part 8 picks up from here, moving from the silicon to regions, quotas, and global endpoints, where getting the chip is one thing and getting it in the region you need is another.

The same accelerator question plays out on AWS with its own custom silicon. The AWS build covers Trainium and Inferentia versus GPUs, and when each wins, which is the closest mirror of this decision on another cloud.

References

• Cloud TPU v6e Trillium documentation
• Ironwood TPUs and new Axion VMs, Google Cloud blog
• A4 VMs powered by NVIDIA B200, Google Cloud blog
• Cloud TPU pricing

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts