My first Bedrock distillation job finished clean. Green status, a shiny custom model ID, and then I tried to call it on demand and got an error. Distilled models do not run on the shared on demand pool. You host them on Provisioned Throughput, and that one fact changes the whole economics of the feature. So before we get to teachers and students, hold onto this: the training is the easy part, the hosting is the decision.
Teacher, student, and synthetic data
Distillation is a way to move the behavior of a big model into a small one for a specific job. You pick a teacher, a large model whose answers you like. You pick a student, a small model from the same family that you want to end up with. You hand Bedrock a set of prompts that look like your real traffic. Bedrock runs those prompts through the teacher, collects the answers, and uses them as labeled training data to fine tune the student. The output is a custom student model that has been pulled toward the teacher’s behavior on your kind of input.
The part people miss is the word synthetic. You are not just replaying teacher answers one for one. Bedrock applies data synthesis on top: it can expand your prompt set with similar prompts, and when you give it prompt and response pairs as golden examples, it generates more high quality responses in that style. That is why a few thousand real prompts can become a fuller training set. It is also why the teacher’s inference bill shows up in your costs, because every synthetic answer is a real teacher call.
Compared with the plain fine tuning from Part 16, the difference is where the labels come from. In ordinary fine tuning you supply the correct answers. In distillation the teacher supplies them. If you do not have thousands of hand labeled answers but you do have a big model that already does the task well, distillation is the cheaper path to a training set.
When distillation beats fine-tuning and RAG
Distillation is not the first tool you reach for. It sits at the end of a short ladder. Start with a good prompt. If the answers are wrong because the model lacks your facts, add retrieval, which is Knowledge Bases from Part 12. If the answers are wrong because the model does not follow your format or tone, plain fine tuning fixes that with your own labels. You reach for distillation in one specific situation: a big model already does the task well, but it is too slow or too expensive to run at your volume, and you want a small model to match it.
The honest test is whether the small base model is already close. Distillation narrows a gap, it does not perform miracles. If Nova Micro out of the box gets you 70% of the way and the teacher gets 95%, distillation can pull the student most of the rest of the way for that narrow task. If the small model is at 30%, you are asking too much and you should either keep the big model or rethink the task. Try the cheap student on your eval set before you spend a dollar on a job.
One more filter. Distillation makes sense when the task is stable and high volume. You are committing to hosting a custom model on reserved capacity, so the traffic has to justify keeping that capacity warm. For a task you run a few hundred times a day, the math rarely works, and I will show you why in the break-even section.
Which teacher and student pairs Bedrock allows
You cannot mix providers. The teacher and student must come from the same model family, because the student inherits the teacher’s tokenizer and architecture assumptions. Amazon to Amazon, Anthropic to Anthropic, Meta to Meta. Here are the pairs that were supported at general availability, which landed in May 2025.
| Provider | Teacher | Student | Job region |
|---|---|---|---|
| Amazon | Nova Premier | Nova Pro, Lite, Micro | us-east-1 |
| Anthropic | Claude 3.5 Sonnet v2 | Claude 3 Haiku | us-west-2 |
| Meta | Llama 3.1 405B, Llama 3.3 70B | Llama 3.1 70B or 8B, Llama 3.2 1B or 3B | us-west-2 |
Two region facts bite people. The Amazon pairs run the job in us-east-1 and you must buy Provisioned Throughput in us-east-1, and Nova custom models cannot be copied to another Region. The Claude and Llama jobs run in us-west-2, but there you can copy the distilled model to another Region and then buy throughput where you need it. If your workload has to live in Frankfurt or Sydney, that copy rule decides which family you can even use. Check the current support list before you commit, since AWS adds pairs over time. [VERIFY exact student list for Claude 3.5 Sonnet v2 beyond Claude 3 Haiku]
How a distillation job runs
Under the hood a distillation job is a model customization job. You submit it with the same control plane call as fine tuning, CreateModelCustomizationJob, with the customization type set to distillation and a config that names the teacher. Bedrock generates teacher responses, synthesizes extra data up to a cap of 15,000 prompt and response pairs, fine tunes the student, and writes the custom model plus metrics to your output S3 bucket. You need an IAM role that Bedrock can assume to read your input and write your output, the same pattern as any customization job.
You do not tune the usual knobs here. There are no epochs or learning rate to set the way you might for plain fine tuning, because Bedrock manages the training recipe for distillation. You control the inputs and the models, not the optimizer. That is a feature, not a limitation, for most teams, but it means you cannot rescue a bad result by twiddling hyperparameters. If the eval is weak, the fix is better prompts or a different teacher, not a knob.
aws bedrock create-model-customization-job
--job-name distill-support-nova-micro
--customization-type DISTILLATION
--base-model-identifier amazon.nova-micro-v1:0
--role-arn arn:aws:iam::111122223333:role/BedrockDistillRole
--output-data-config s3Uri=s3://my-bkt/distill-out/
--training-data-config s3Uri=s3://my-bkt/prompts.jsonl
--custom-model-name support-nova-micro-distilled
--customization-config '{"distillationConfig":{"teacherModelConfig":{"teacherModelIdentifier":"amazon.nova-premier-v1:0","maxResponseLengthForInference":1024}}}'
--region us-east-1Expected output: the call returns a jobArn. Track it with get-model-customization-job until status reads Completed, then the custom model appears under your custom models.
Failure mode: the most common early failure is an IAM role that cannot read the input bucket or write the output prefix, which surfaces as a Failed status with an access denied message, not an API error at submit time. The nested field names above are the shape you pass; confirm them against the current API reference before you script this. [VERIFY exact customizationConfig field names]
Two ways to supply training data
You have two front doors for the prompts. The first is a JSONL file in S3, one prompt per line, optionally with a golden response. That is the clean path when you are building a new task from scratch. The second is more interesting for anyone already in production: Bedrock can read your historical invocation logs and use real traffic as the distillation input. If you have model invocation logging turned on, the prompts your users already sent become your training set, filtered by metadata you attach to requests.
I reach for invocation logs when I have been running the teacher in production for a while. The distribution is honest, because it is literally what users asked. A hand written prompt file always drifts toward the cases you imagined rather than the ones you got. The trade is privacy and governance: those logs may hold customer data, so the same residency and KMS rules from Part 10 apply to the training set, not just the runtime. Either way the cap is 15,000 pairs, so more is not always the goal. A few thousand representative prompts usually beat a noisy dump.
Provisioned throughput, the hidden bill
Here is the fact that sank my first project plan. A distilled model is a custom model, and custom models do not run on the on demand token pool. To serve one you buy Provisioned Throughput, which is capacity measured in Model Units and billed by the hour whether or not a single request comes in. The per hour rate matches the base student model. So the 75% per token saving is real, but it only shows up if you keep the reserved capacity busy enough that the hourly cost divided by your request count lands below the on demand price of the small base model.
Read that again, because it inverts the usual pitch. Distillation does not automatically save money. It saves money at volume. At low or spiky volume you are paying for idle Model Units, and calling the plain small base model on demand is cheaper. The three cost components below are the whole story.
| Cost component | When you pay | Priced as |
|---|---|---|
| Synthetic data generation | Once, during the job | Teacher on-demand token rate |
| Student training | Once, during the job | Customization training rate |
| Hosting | Every hour, until you delete it | Provisioned Throughput per Model Unit |
A break-even you should run first
Before any job, model the crossover. On demand cost rises with every request. Provisioned Throughput is a flat monthly floor once you commit a Model Unit, roughly the hourly rate times about 730 hours a month, and it barely moves as traffic grows until you saturate the unit. Plot both and you get one intersection. Below it, on demand wins. Above it, the distilled model on reserved capacity wins. Your only job is to honestly estimate where your steady traffic sits relative to that line.
Worked example
Say a support classifier runs the teacher at 2 million tokens a day. Move it to a distilled student and per token cost drops about 75%, so the token bill for that traffic falls to roughly a quarter. But you now hold one Model Unit of Provisioned Throughput, a fixed monthly floor of the hourly rate times about 730 hours. If that floor is larger than your entire old on-demand token bill, distillation loses money no matter how good the accuracy is. The classifier only wins once daily volume is high enough that the reserved unit runs near full and the saved token cost clears the hourly floor. Model your own token rate against the current Provisioned Throughput price before you run the job. [VERIFY current Model Unit hourly rate for the chosen student]
Where distillation goes wrong in production
Three failures show up again and again. The first is the idle unit I keep hammering: teams distill a low volume task, forget the hourly floor, and get a bigger bill than before. The second is a weak teacher. If your teacher only reaches 80% on the eval, the student will land under that, so a distilled model can quietly cap your quality lower than you expected. Test the teacher on your real eval set first, and treat its score as your ceiling.
The third is drift. A distilled student is frozen to the task it learned. When your traffic shifts, a new product line, a new question shape, the student was never trained on it and degrades faster than the general purpose base model would. Plan to re-distill on fresh invocation logs on a schedule, and keep the eval set current so you notice the day the student falls behind. Distillation is not a one time job, it is a small recurring pipeline.
Buy the distillation only when you can keep it busy
My rule is simple. Distill when three things are all true: the teacher already does the task well on your eval, the small base model is already close, and the traffic is high and steady enough to keep a Model Unit warm. Miss any one and you are better off calling the base model on demand or staying on the teacher. Distillation is a volume play dressed up as an accuracy feature, and the volume half is where projects fail.
Next in the series we move from customizing single models to giving models a place to learn from your own examples at scale, with SageMaker JumpStart in Part 18. If you build across clouds, the same idea shows up as distillation and stored completions on Azure and as tuning on Vertex, and the vendor-neutral view of why data, not model size, decides quality is the principle behind every distillation that works. This week, before you plan a job, pull one day of invocation logs, count the tokens, and multiply the small model on-demand rate by your monthly volume. If that number is smaller than one Model Unit of Provisioned Throughput, you have your answer, and you just saved yourself a job.
References
• Customize a model with distillation in Amazon Bedrock
• Amazon Bedrock Model Distillation is now generally available
• Supported models and Regions for Model Distillation
• Amazon Bedrock pricing
About The Author
DrJha