Your Gemini 2.5 Flash-Lite deployment answers the easy 85 percent of tickets as well as Pro would, and it fumbles the hard 15 percent. You have two moves. Route the hard ones to Pro and pay the premium on every one of them, or teach Flash-Lite to handle them itself. This part is about the second move, and the reason it is cheaper than it sounds is that you never label the hard cases by hand. A bigger model does it for you.

Distillation is the technique that pulls this off. It trains a small model on the outputs of a large one instead of on examples a person wrote. In Part 16 you tuned Gemini on hand-labelled pairs. Here the labels come from the teacher, which changes the economics of the whole job. I will walk through what distillation changes in the student, when it beats plain fine-tuning, how a job is wired on Vertex AI, what the teacher calls cost, a real run in code, and the places it quietly lets you down.

What distillation actually does

Distillation trains a small student model to reproduce the behaviour of a large teacher model. You pick a capable model as the teacher, say Gemini 2.5 Pro, and a cheap model as the student, say Gemini 2.5 Flash or Flash-Lite. You feed the teacher a pile of prompts, it produces answers, and the student trains on those answers until it responds the same way. On Vertex AI, Google Cloud managed platform for building and running models, this runs as the Gemini Distillation Service, a tuning method you pick rather than a separate product.

Here is the part that separates it from ordinary fine-tuning. Supervised tuning learns from the final text of each labelled answer and nothing else. Distillation on Vertex AI copies more than the final text. It uses the teacher completions and the teacher reasoning paths, the internal chain the teacher worked through before it answered. The student learns not just what the teacher said but a trace of how it got there, which is why a distilled Flash model can pick up reasoning behaviour that a plain fine-tune on final answers alone tends to miss. That is the whole pitch: reasoning transfer without you paying a person to write out the reasoning.

Say that plainly. You are moving a slice of a large model behaviour into a small one, so you can run the small one in production and stop paying large-model prices on every call. The ceiling is the teacher. The student is trying to imitate it, so on the tuned task the student climbs toward the teacher quality and does not pass it. If the teacher gets your task wrong, the student learns the wrong thing faithfully.

Distillation or supervised fine-tuning, which one?

Both methods end at the same place, a tuned small model billed at the small model rate. The split is where the labels come from. Supervised fine-tuning needs you to supply the right answers, which means someone builds a dataset by hand or from historical records. Distillation needs prompts only for training, because the teacher supplies the answers during the run. So the honest decision rule is about your data, not your ambition.

A useful way to hold it: fine-tuning teaches a model your answers, distillation teaches a small model another model answers. When you already pay Pro to get a task right and the only problem is the bill, distillation is the direct fix. When you need behaviour no available model produces yet, distillation has nothing to copy and you are back to writing labels.

flowchart LR
  P[Prompt only dataset] --> T[Teacher Gemini Pro]
  T --> A[Answers plus reasoning]
  A --> S[Train student Flash Lite]
  V[Validation set with answers] --> S
  S --> M[Tuned student endpoint]
  M --> D{Beats baseline}
  D -->|yes| Prod[Serve at Flash price]
  D -->|no| P

How Vertex AI wires a distillation job

Four things go into a job. A student model, the small one you want to end up with. A teacher model, the large one whose behaviour you are copying. A training dataset of prompts in a Cloud Storage bucket, JSONL, one prompt per line, no answers needed because the teacher fills them in during the run. And a validation dataset that does carry ground-truth answers, because Vertex AI uses it to score the student as it trains, and a validation set with no answers cannot score anything.

The asymmetry trips people up, so hold onto it. Training data is prompts only. Validation data is prompts with answers. During the run Vertex AI sends each training prompt to the teacher, collects the answer and the reasoning, and trains the student on that. It never has to ask you for a training label. Below is what a couple of training lines look like. Note there are no model turns, only the user prompt.

{"contents":[{"role":"user","parts":[{"text":"Classify this ticket priority and explain why: card declined twice, customer travelling tomorrow."}]}]}
{"contents":[{"role":"user","parts":[{"text":"Classify this ticket priority and explain why: password reset link expired."}]}]}

Expected shape: each training line is a bare prompt with no answer. The teacher generates the priority and the reasoning at run time. Failure mode: add a model turn to a training line and you are back to supervised tuning on that line, not distillation, so keep training prompts answer-free and put your ground-truth only in the validation file.

Pick your student and teacher from the same Gemini generation to keep tokenisation and formatting aligned. A common pairing is Gemini 2.5 Pro as teacher and Gemini 2.5 Flash-Lite as student, which is the widest quality-to-cost gap in the 2.5 line and therefore the biggest saving if the student can close it. Confirm the current list of models each side supports for distillation in the console before you commit, since the supported pairings shift as new models land.

What distillation costs, teacher calls included

Three costs, and only one of them recurs. First, the teacher inference you pay for during training, because generating an answer for every training prompt is a real batch of Pro calls billed at Pro rates. Second, the training charge itself, billed on the tokens or characters processed. Third, the recurring one, student inference, and here is the good news that makes the whole thing worth it: the distilled student serves at the student model rate with no distillation surcharge. You pay the teacher once, to teach, and then you never call it again.

Run a distillation job

With both files in a bucket, the job is a single call in the Google Gen AI SDK for Python. You name the student as the base model, name the teacher, set the method to distillation, and point it at your prompts. Vertex AI runs the teacher, trains the student, and hands back a tuned model you call exactly like a base Gemini model.

from google import genai
from google.genai import types

client = genai.Client(vertexai=True, project='my-proj', location='us-central1')

job = client.tunings.tune(
    base_model='gemini-2.5-flash-lite',
    training_dataset=types.TuningDataset(
        gcs_uri='gs://my-bucket/distil/prompts.jsonl',
    ),
    config=types.CreateTuningJobConfig(
        method='DISTILLATION',
        base_teacher_model='gemini-2.5-pro',
        epoch_count=3,
        validation_dataset=types.TuningValidationDataset(
            gcs_uri='gs://my-bucket/distil/val.jsonl',
        ),
        tuned_model_display_name='ticket-priority-distil-v1',
    ),
)

print('job:', job.name)
print('state:', job.state)

Expected output: a tuning job resource name and a running state, then, once it finishes, a tuned model you point production at. Failure mode: a validation error usually means your training file smuggled in answer turns or your validation file left them out, since distillation wants the opposite of each. A quota error means you are over the concurrent tuning limit, which starts low. Confirm the current SDK argument names against the distillation docs before you run, since the tuning API has changed shape more than once.

When it finishes you score the student on the held-out set, and only if it clears your bar do you route traffic to it. Calling the tuned student is the same code from Part 11, with the tuned endpoint in place of the base model id, so nothing downstream in your app has to change.

Where distillation disappoints

The first disappointment is the ceiling. A student imitates a teacher, so it approaches teacher quality on the tuned task and stops. Google frames the realistic outcome as the student capturing a large share of the teacher quality at a fraction of the cost, not matching it outright. If you needed to beat the teacher, distillation is the wrong tool, and you are looking at a better base model or a different approach entirely.

The second is teacher cost on a big dataset. Every training prompt is a paid Pro call, so a distillation set of a hundred thousand prompts is a hundred thousand Pro inferences before you have trained anything. On a large corpus that one-time charge stops being a rounding error, and you should price it before you launch the job, not after. The third is the same maintenance tail every tuned model carries. The student is frozen at the day you trained it, the base models keep improving under it, and when your task drifts you retrain. Distillation does not escape that, it just changes who writes the labels for the retrain.

Reach for distillation when labels are the bottleneck

My rule is short. If a large model already does your task well and the only problem is what it costs at volume, distil it into a small one and serve the small one. That is the case where distillation pays for itself fast, because you skip the labelling entirely and the recurring bill drops to the student rate. If you do not have a teacher that gets the task right, distillation has nothing worth copying, and you belong back in supervised fine-tuning from Part 16 or a stronger base model instead.

Before you run a full job, do one thing: send a few hundred of your real prompts through the teacher, score the answers by hand, and confirm the teacher clears the bar you plan to hold the student to. That single check is the difference between distilling quality and distilling a bug at scale. If the teacher passes, you have a training set already, and the run itself is the easy part. Open the console, point a distillation job at those prompts, and measure the student before you promise anyone a saving.

References

• Gemini Distillation Service, Google Cloud documentation
• Distillation notebook, GoogleCloudPlatform vertex-ai-samples
• Generative AI on Vertex AI pricing

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts