Pull up your Azure OpenAI bill for one narrow task, say classifying support tickets, and you are almost certainly paying gpt-4.1 rates for work a model a tenth the size could handle. The catch was always getting the small model to answer like the big one. Stored completions close that gap. Add store=True on the same call you already make, and Azure keeps every request and response as labeled data you can later hand to a smaller model. That is distillation, and it starts with one flag.

What a stored completion is

A stored completion is an ordinary chat completion that Azure keeps a copy of. You make the same call you always make, with one extra parameter, store=True. Azure records the input messages and the model output and files them under Stored completions in the Foundry portal. A second parameter, metadata, lets you attach your own key and value tags to each record, a task name, an environment, a prompt version, so you can slice the pile later instead of drowning in it. Nothing about the response your app receives changes. The only difference is that the exchange is now saved.

Why keep them at all? Because two later jobs need real examples of your task, and typing those by hand is the worst part of fine-tuning. Stored completions turn live production traffic into that dataset for free. Evaluation reads them to score how your model does on real inputs, and distillation reads them to train a smaller model. Azure enabled stored completions and evaluation in every Azure OpenAI region, so this is not gated to a handful of locations. One caution worth checking before you rely on it: captured records have a retention window and are meant as a working buffer, not a permanent archive, so export what you need. [VERIFY: current stored completions retention period on Azure]

Why capture the traffic you already pay for

Most teams reach for fine-tuning and immediately hit the data problem. You need hundreds of clean prompt and answer pairs, and hand-writing them is slow and tends to miss the weird inputs real users send. Stored completions solve this by letting the expensive model you are already running generate the dataset as it works. Every ticket your gpt-4.1 deployment classifies today, with store=True set, becomes a labeled example tomorrow. The teacher is doing double duty: serving production and quietly writing your training set.

There is a quality angle too. Distillation trains the student on the teacher outputs, so the student inherits the teacher behavior on your exact inputs, not on some generic benchmark. If your gpt-4.1 prompt has been tuned over months to answer in the right format and tone, those months of prompt work are baked into every captured answer, and the student learns them. This is why a distilled small model often beats the same small model fine-tuned on hand-written data. The examples came from a stronger model doing the real job.

How distillation makes a smaller model behave like a big one

Distillation here is not a separate algorithm you configure. It is supervised fine-tuning, the SFT method from Part 16, where the training answers happen to come from a bigger model instead of a human. The teacher is a large capable model, gpt-4.1 say. The student is a small cheap model you can afford to run hot, gpt-4.1-mini or gpt-4.1-nano. You collect the teacher answers as stored completions, then run an SFT job that trains the student to reproduce them. The student ends up specialized on your one task, and it is far cheaper per token than the teacher.

What you trade away is generality. A distilled nano is very good at the task you distilled and no better than stock nano at anything else. That is the point, not a flaw. You are not trying to build a general assistant, you are trying to make one narrow behavior cheap. The same pattern runs on AWS, where Bedrock exposes distillation as its own managed flow rather than folding it into fine-tuning, and I walk that version in the Bedrock distillation part. Azure keeps it closer to plain fine-tuning, which means fewer new concepts once you know SFT.

Model names and their fine-tune eligibility move, so confirm the current student options before you plan a run. [VERIFY: gpt-4.1-mini and gpt-4.1-nano fine-tune availability]

From captured calls to a fine-tuned student

The workflow has three stages, and the middle one is the one people skip and regret. Stage one, capture: run the teacher with store=True until you have a meaningful pile of real exchanges. Stage two, curate: in the Stored completions view you filter by your metadata tags, drop the bad answers, and optionally push the set through Evaluation to score quality so you keep only high-scoring examples. Stage three, distill: export the curated set and start an SFT job with the student as the base model. A job needs at least 10 examples to run, but 10 teaches the student almost nothing. Treat a few hundred as a floor and low thousands as the target.

Here is the capture step in code. It is your normal call with two additions, and a small loop that would build the dataset as traffic flows.

from openai import AzureOpenAI

client = AzureOpenAI(api_version='2025-04-01-preview')

def classify(ticket_text):
    resp = client.chat.completions.create(
        model='gpt-4.1',            # the teacher
        store=True,                 # keep this exchange
        metadata={'task': 'ticket-triage', 'promptver': 'v3'},
        messages=[
            {'role': 'system', 'content': 'Classify the ticket. Reply: category | priority.'},
            {'role': 'user', 'content': ticket_text},
        ],
    )
    return resp.choices[0].message.content

print(classify('my card was charged twice'))

Expected output: a line like billing | P2, and the exchange now appears under Stored completions in the portal tagged task=ticket-triage. Failure mode: leave store=True off and everything runs fine but nothing is captured, so weeks later you have no dataset. Metadata values must be strings and the number of pairs per record is capped, so keep tags short. [VERIFY: metadata pair limit and current api_version]

When distillation actually saves money

This is where most write-ups stop short. A distilled student is cheap per token, but a fine-tuned model on Azure carries an hourly hosting charge, billed whether or not it serves a single request, the same meter I flagged in Part 16. So distillation is not free money. It only wins once your teacher inference spend is large enough to cover that hosting floor plus the student token cost. Below that line, keeping the teacher is cheaper, and a stubborn team distills anyway and pays more.

The practical rule I use: distill when the teacher is your top line item and traffic is steady, skip it when volume is spiky or small. If you cannot keep a hosted student busy, its hourly meter eats the savings. Reserved or high-throughput hosting changes the arithmetic, so price the deployment option you would actually use, not the cheapest one on the page.

Curating and scoring before you spend a token

The student copies whatever you feed it, good answers and bad ones alike. A teacher is not right every time, so an unfiltered capture set trains your student on the teacher mistakes as faithfully as its wins. This is why the curate stage matters more than the training settings. In the Stored completions view you filter to the task and version you want, then either eyeball and delete the poor ones or, at any real scale, push the set through Evaluation and keep only records that score above a threshold. That scored subset is your training file.

Coverage matters as much as quality. If ninety percent of your captures are easy tickets and the hard categories barely appear, the student will be excellent on easy inputs and weak exactly where you needed help. Before training, count examples per category and deliberately keep more of the rare, hard ones rather than a random slice. A balanced few hundred beats a lopsided few thousand. Hold back a portion the training never sees, so you can score the student against the teacher on fresh inputs afterward.

Where distillation quietly goes wrong

Three failures show up again and again. The first is distilling a task that never needed it: if the gap was missing facts, no amount of imitation helps, because the teacher was guessing too, and that belongs in retrieval, not weights. The second is drift. The teacher answers you captured last quarter describe last quarter behavior, and if your task or policy shifted, a student trained on stale captures ships stale behavior confidently. Recapture and retrain on a schedule rather than distilling once and forgetting. The third is silent quality loss on the long tail: the student matches the teacher on common inputs and falls apart on rare ones you had too few examples of.

The defense against all three is the held-out set. Score the student against the teacher on inputs neither the training nor your eyeballs have seen, broken out by category so the long-tail weakness cannot hide inside a good average. If the student trails the teacher by more than you can accept on the categories that matter, add captures where it is weak and retrain, or accept a larger student. Distillation is a loop you tighten, not a switch you flip once. Next in the series, Part 18 moves from Azure OpenAI models to open weight models on Azure Machine Learning, which is where you go when you want to own the student weights outright.

Turn on capture now, distill when the math clears

My recommendation splits into two moves with very different urgency. Turn on store=True with sensible metadata today, on every task where a smaller model might one day do the job. Capture is nearly free, it does not change what your app returns, and the dataset it builds is the expensive part of any future distillation. There is no reason to wait. Distilling is the second move, and it waits for the numbers: only pull the trigger once the teacher is a real line item, traffic is steady enough to keep a hosted student busy, and your held-out scores say the student holds the task. When those line up, distillation gives you a model that is roughly twenty times cheaper per token and faster, on the one behavior you care about.

When not to bother: spiky or low volume, a task that is really a retrieval problem, or a teacher prompt still changing week to week, because you would be distilling a moving target. Your next step is small and costs nothing. Add store=True and a metadata task tag to your highest-volume Azure OpenAI call this week, and let the captures pile up while you decide.

References

• Stored completions and distillation, Microsoft Foundry docs
• Azure OpenAI Service pricing
• New fine-tuning models and techniques in Azure AI Foundry

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts