A team I advise watched their Gemini Flash bill climb every week, not because any single request was expensive but because a background job was quietly re-summarising the same corpus millions of times a day. The fix was not a cheaper prompt. It was moving that one high-volume, low-variety workload onto a model they host themselves. That model was Gemma, and this part is about when that move is right and how you actually make it on Vertex AI.
TL;DR
Gemma is Google’s family of open-weight models you download, fine-tune, and run on your own Vertex AI endpoints, as opposed to Gemini, which you call as a managed service. In 2026 the current text lineup is Gemma 3 at 270M, 1B, 4B, 12B, and 27B, and Gemma 4 at E2B, E4B, 26B A4B, and 31B, plus specialist variants like PaliGemma, EmbeddingGemma, ShieldGemma, and MedGemma.
Self-hosting Gemma only beats calling Gemini once your volume is high and steady. Below that line the managed model is cheaper and far less work. I show the crossover with real numbers further down.
What Gemma actually is
Gemma is a set of open-weight models built by Google DeepMind from the same research that produced Gemini, but shipped as files you can download. Open weight means you get the trained parameters and can run them anywhere: your laptop, a Vertex AI endpoint, a GKE cluster, or a phone. That is the whole difference from Gemini. Gemini lives behind Google’s API and you never touch its weights. Gemma lands in your bucket and runs on hardware you rent and manage.
Why does anyone want that when the managed model is easier? Three reasons keep coming up in real projects. You want the model inside your own VPC with no request ever leaving, for a regulator or a security team. You have a narrow task at very high volume where a small fine-tuned model is both cheaper and faster than a large general one. Or you need to run at the edge or offline, where an API is not an option. If none of those describe you, Gemini is the better default and this part is a tour rather than a prescription.
One thing Gemma is not: a drop-in Gemini replacement. The 27B model is capable, but it is not going to match the largest Gemini Pro on hard reasoning. You trade some ceiling for control and unit economics. Know which of those you are buying.
The Gemma lineup in 2026
The family grew fast and the names run together, so here is the map. The core text models are what you reach for most. Gemma 3 shipped in March 2025 at 1B, 4B, 12B, and 27B, with a tiny 270M added in August 2025 for on-device and hyper-narrow tasks. Gemma 3 handles a 128K token context, understands more than 140 languages, and was designed to run on a single GPU or TPU. Gemma 4 arrived on 31 March 2026 in E2B, E4B, 26B A4B, and 31B sizes, then a 12B Unified variant on 3 June 2026, adding text, audio, and image input and a context window up to 256K tokens.
Around the core sit the specialists, and you pick these by task rather than size. PaliGemma is vision plus language for captioning and visual question answering. EmbeddingGemma at 308M produces retrieval embeddings and is what you would pair with the RAG stack from Part 12. ShieldGemma is a safety classifier that scores content against harm categories. MedGemma, TranslateGemma, CodeGemma, and RecurrentGemma target health, translation, code, and long-sequence efficiency. VaultGemma is trained with differential privacy. You do not need to know all of them; you need to know that when someone says Gemma, ask which one.
| Model | Sizes | Input | Context | Use it for |
|---|---|---|---|---|
| Gemma 3 | 270M, 1B, 4B, 12B, 27B | Text, image | 128K | General text on one GPU |
| Gemma 4 | E2B, E4B, 26B A4B, 31B, 12B Unified | Text, audio, image | Up to 256K | Current multimodal work |
| Gemma 3n | E2B, E4B | Text, image, audio | 32K | Phones and edge |
| EmbeddingGemma | 308M | Text | 2K | RAG embeddings |
| PaliGemma 2 | 3B, 10B, 28B | Text, image | Task dependent | Vision question answering |
Gemma or Gemini? Where self-hosting pays
This is the decision that matters, so I will be blunt. A managed Gemini call bills per token and costs nothing when idle. A Gemma endpoint bills for GPU time by the hour, running or not, because the accelerators are reserved for you. So the two cost curves cross. Below some volume, per-token Gemini is cheaper. Above it, the fixed hourly cost of your own endpoint wins because you have spread it across enough requests.
The chart uses round assumed rates, not a quoted benchmark. The shape is the point. If your traffic is spiky or low, an idle Gemma endpoint burns money for nothing while Gemini would have charged you zero. That is the single most common mistake I see: a team stands up a Gemma endpoint for a workload that runs a few thousand times a day, then wonders why the bill went up. For the current Gemini token rates that anchor the red line, see the pricing mechanics in Part 6.
How deployment works on Vertex AI
You have three main paths onto Vertex, in rising order of control and effort. The one-click path is Model Garden: find the Gemma card, click deploy, and Vertex spins up a managed endpoint backed by a prebuilt vLLM container. vLLM is a serving engine that batches requests and manages the KV cache so a GPU stays busy. The second path is the same deploy from the command line or SDK, which is what you put in a pipeline. The third path is GKE, where you run the vLLM container yourself on a cluster you own, trading convenience for full control over autoscaling and networking.
flowchart LR A[Model Garden Gemma card] --> B[Deploy with vLLM container] B --> C[Vertex AI Endpoint] C --> D[GPU nodes autoscale] C --> E[Your app calls predict] A --> F[Or run vLLM on GKE yourself] F --> C
Whichever path you take, the model itself is the same file. The choice is about who runs the container and how much you want to tune the plumbing. For a first deployment, use Model Garden and let Vertex pick sane defaults. Move to GKE only when you hit a wall the managed endpoint cannot clear, such as a custom autoscaling policy or a shared cluster you already operate.
# 1. List Gemma models you can self deploy from Model Garden
gcloud ai model-garden models list --model-filter=gemma
# 2. Deploy Gemma 3 27B instruction tuned to a managed endpoint
gcloud ai model-garden models deploy
--model=google/gemma3@gemma-3-27b-it
--region=us-central1
--machine-type=g2-standard-24
--accelerator-type=NVIDIA_L4
--accelerator-count=2
# 3. Call the endpoint once it is ready
curl -X POST
-H "Authorization: Bearer $(gcloud auth print-access-token)"
-H "Content-Type: application/json"
https://us-central1-aiplatform.googleapis.com/v1/ENDPOINT_RESOURCE:predict
-d '{"instances":[{"prompt":"Summarise this ticket in one line."}]}'Expected: step 1 prints model IDs with a self-deploy flag; step 2 returns a long-running operation then an endpoint resource name; step 3 returns a JSON prediction. Common failure: a 429 quota error on step 2 because your project has zero L4 accelerators in that region. Fix by requesting accelerator quota, or pick a region where you already have it.
Sizing a Gemma endpoint
Sizing comes down to fitting the weights in GPU memory with room for the KV cache, then buying enough throughput. A rough rule for weights: parameters times two bytes for 16-bit precision, so a 27B model needs about 54 GB just to load, before any cache. That does not fit on one 24 GB L4, which is why the deploy above asked for two. Quantise to 8-bit and the weights roughly halve, often letting a smaller box carry the model at some quality cost you should measure, not assume.
Worked example
Say the summarisation job runs 40 requests per second at peak, each about 600 input and 120 output tokens. At an assumed 1,500 output tokens per second per L4 for the 27B model, one GPU serves roughly 12 concurrent requests before latency climbs. To hold 40 per second with headroom you provision three to four L4s across two nodes.
Monthly floor at an assumed $0.70 per L4 hour is four times 0.70 times 730, about $2,044 for the accelerators alone. That same volume, near 1.2 billion tokens a month, sits just past the crossover in Figure 2, so self-hosting is defensible here. Halve the volume and it is not.
Two habits save you here. Load test with your real prompt shapes, because a 600-token input behaves nothing like a 6,000-token one for cache pressure. And set the endpoint to scale to zero if your platform allows it and the workload is bursty, so you stop paying for idle GPUs. If it cannot scale to zero, size for your average load and let a queue absorb spikes rather than paying for peak all month.
Licensing changed with Gemma 4
This one matters to legal review and is easy to miss. Gemma 1 through 3 ship under the custom Gemma Terms of Use, which grant broad commercial rights but attach a prohibited-use policy and downstream conditions. Gemma 4 moved to the Apache 2.0 license, a standard permissive open-source license your counsel already knows. If your organisation balked at the custom terms for earlier Gemma, Gemma 4 removes that objection. If you are still on Gemma 3, you are on the older terms, so do not assume the two are interchangeable on paper.
Practical advice: pin the exact model and its license text in your model registry alongside the weights, and record which license each deployed endpoint runs under. When an auditor asks what governs the model answering customer queries, you want a one-line answer, not an afternoon of archaeology.
Where Gemma endpoints bite you
The second trap is quota. Accelerators like L4 and A100 are quota-limited per region and often start at zero in a fresh project, so your first deploy fails with a 429 and a wait for approval. Request the quota days before you need it, not the morning of a demo. The third is capability creep: someone benchmarks Gemma 27B against Gemini Pro on a hard reasoning set, sees a gap, and blames the deployment. The deployment is fine. You picked a smaller model on purpose. Compare Gemma against the right baseline, which is a similarly sized model or your own fine-tuned task metric, not the frontier.
If you want to push a general Gemma toward your task rather than just serving it, that is fine-tuning, and the supervised tuning workflow from Part 16 applies to open weights too, with the difference that you own and host the resulting checkpoint. The Azure side of this same open-versus-managed choice, if you run both clouds, is covered in the Azure open models part.
Start small, self-host only the workload that earns it
Here is where I would start. Keep your general traffic on Gemini. Find the one workload that is high volume, low variety, and either privacy-bound or clearly cheaper self-hosted, and move only that onto a Gemma endpoint. Deploy the smallest model that clears your quality bar, size it from a real load test, and put a delete-by tag and a budget alert on it the same hour you create it. If Gemma 4 fits your task, take it for the Apache 2.0 license alone. Then measure the bill for a full month against what Gemini would have charged, and keep the endpoint only if the numbers still favour it.
Next in the series we leave serving behind and go to training at scale: TPU pods and GKE, where you are not renting one endpoint but orchestrating a cluster. Pick the one workload in your stack that might belong on Gemma, and cost it against Gemini before you deploy anything.
References
- Gemma releases, Google AI for Developers
- Use the new Gemma 3 on Vertex AI, Google Cloud Blog
- Use Gemma open models, Vertex AI documentation
- Gemma 4 available on Google Cloud, Google Cloud Blog
About The Author
DrJha