Halfway through phase two, a card wedged and training aborted. I re-ran the command, saw a prompt asking whether to start training from the beginning, read it too fast, typed y, and watched it clear about five hours of phase one checkpoints. That one keystroke is why this part spends as much time on recovering a run as on starting one.

Where the assistant stands before tuning

Last part the support assistant produced a dataset and nothing more: a validated taxonomy leaf for the company product docs, and two files written by generation, knowledge_train_msgs and skills_train_msgs, sitting in the datasets directory. This part feeds those two files to training and gets back a Granite model that answers questions about Acme Relay instead of guessing. Tuning a base model on your own data is the fine tuning arm of the choice the GenAI series frames under fine tuning vs RAG vs prompting, and here we commit to it.

One term before the commands. Multi phase alignment means the LAB training strategy runs in two ordered stages, knowledge then skills, with a checkpoint carried from the first stage into the second, rather than mixing both kinds of data into a single pass. That ordering is deliberate, and getting it wrong by training skills only, or by feeding the files in the wrong slots, produces a fluent model that has not actually learned your facts.

Why fine tune at all when retrieval is cheaper to run and easier to change? For this assistant the payoff is grounding and latency together. Retrieval, which the assistant gets in a later part, pulls the right passage at request time; tuning bakes the product vocabulary and the shape of a good support answer into the weights, so the model does not fumble Acme Relay terminology before retrieval even lands. Both are complements, not rivals, and this part builds the tuned half.

Multi phase training and what each phase does

Phase one is knowledge training. It runs several epochs over the knowledge JSONL, saving a checkpoint each epoch, and InstructLab promotes a late knowledge checkpoint, the seventh epoch in the default recipe, as the starting weights for phase two. Phase two is skills training, which teaches the model to use those facts in the shapes a user asks for. After phase two, every checkpoint it produced is scored with MT-Bench and the best scoring one is handed back as your model. Keep the table below as the reference for which file goes where and what each phase writes.

Two numbers in that recipe explain the shape of a run. Effective batch size jumps from 128 in phase one to 3840 in phase two because skills examples are short and the trainer packs many into a step, while knowledge examples carry long context and must stay small. Learning rate holds at 2e-05 across both, and each epoch writes a checkpoint so nothing is lost if a later epoch degrades. Knowledge runs first for a plain reason: skills training assumes the facts are already present, so teaching format before content leaves a model good at sounding right and bad at being right.

Promotion between phases is easy to miss and worth understanding. Phase one does not hand its final epoch to phase two; the LAB recipe promotes an earlier knowledge checkpoint, the seventh epoch by default, because the last knowledge epochs start to overfit the facts and make poorer starting weights for skills. So overfitting is guarded twice, once when phase one promotes and once when MT-Bench selects at the end. Both guards protect against a model that memorised its training rows.

LAB tunes the full weights rather than a LoRA adapter by default, and that is a deliberate trade. A full fine tune moves the whole model, which is why it needs the four card node, but it also changes behaviour more deeply than a low rank adapter can, which matters when you are teaching facts and not just a style. You can switch to a LoRA run to fit smaller hardware, and later parts weigh that trade for serving, but for a first pass at teaching product knowledge the full tune is the honest baseline to measure against.

A naming detail trips people on a first run: the model you tune is granite-7b-starter, a base model built specifically as a LAB student, not the granite-3.1-8b-instruct you served in Part 8. Starter models are deliberately under aligned so the LAB phases can shape them, and pointing training at an already instruction tuned model fights the alignment it carries and tends to score worse. Tune the starter, then serve the result.

flowchart LR
  A[Knowledge jsonl] --> B[Phase 1 knowledge training]
  B --> C[Promote late knowledge checkpoint]
  C --> D[Phase 2 skills training]
  E[Skills jsonl] --> D
  D --> F[MT-Bench scores every checkpoint]
  F --> G[Best checkpoint is your model]

Running lab-multiphase training

One command runs both phases and the selection. Point phase one at the knowledge file and phase two at the skills file, both from the dated datasets directory generation wrote in Part 9. Add the serving output flag when you want to watch the logs scroll rather than stare at a spinner.

# RHEL AI 1.5, bundled InstructLab, student granite-7b-starter, judge prometheus-8x7b-v2-0
$ ilab model train --strategy lab-multiphase 
    --phased-phase1-data ~/.local/share/instructlab/datasets/2026-07-28_205421/knowledge_train_msgs_2026-07-28T20_54_21.jsonl 
    --phased-phase2-data ~/.local/share/instructlab/datasets/2026-07-28_205421/skills_train_msgs_2026-07-28T20_54_21.jsonl 
    --enable-serving-output

Training Phase 1/2...
TrainingArgs for current phase: model_path=.../granite-7b-starter
  max_seq_len=4096, num_epochs=7, effective_batch_size=128,
  learning_rate=2e-05, warmup_steps=25, checkpoint_at_epoch=True

Phase two starts from the promoted checkpoint, not from granite-7b-starter, and runs at a much larger effective batch size because skills data is short and plentiful. When it finishes, RHEL AI prints the winning checkpoint and its MT-Bench score. Write that path down; serving and evaluation both need it.

Training Phase 2/2...
TrainingArgs for current phase: model_path=.../phase1/checkpoints/hf_format/samples_52096
  max_seq_len=4096, num_epochs=10, effective_batch_size=3840, learning_rate=2e-05

MT-Bench evaluation for Phase 2...
INFO instructlab.model.backends.vllm: vLLM engine successfully started

Training finished! Best final checkpoint: samples_1945 with score: 6.813759384

On our four card L40S node the full run, both phases plus scoring, took a little over nine hours for a taxonomy of eight knowledge leaves. RHEL AI 1.5 added long context training modes that lengthen this further; if you do not need them, setting max_seq_len to 10000 in config.yaml trims the time back. Run it detached and check on it later rather than holding a terminal open for nine hours.

Detached runs are the norm at this length. Add the detach flag and training returns your shell, then ilab process list shows the run and its log path, so you can tail progress or walk away. Everything the trainer uses, learning rate, epoch counts, batch length and max_seq_len, comes from the train block of config.yaml, so tuning those is an edit to one file rather than a wall of flags. Leave them at the RHEL AI defaults for a first run; they are chosen for the LAB recipe, and changing them blindly is how a run that would have scored 6.8 comes back at 6.2.

By default training uses every accelerator it can see on the node. On a shared box, pin it with the gpus setting in config.yaml or a device visibility variable, so a training run does not collide with another job, because two full fine tunes on the same four cards is an immediate out of memory for both. Isolation here is cheaper than the debugging session that follows a silent collision.

InstructLab also offers a skills only strategy that skips phase one entirely. It is tempting when you only wrote skills, and it is the wrong default for a knowledge assistant: without the knowledge phase the model never grounds on your facts, so it learns to answer in the right format about things it does not know. Reach for skills only when you are teaching a capability with no new facts, such as a rewriting or extraction behaviour, and keep lab-multiphase for anything that must be correct about your product.

Reading checkpoints and why the last one is not your model

Phase two saves many checkpoints under the phase directory, each named samples_ plus the number of data points it has seen. List them and you get a spread, not a single answer.

$ ls ~/.local/share/instructlab/phase/phase2/checkpoints/
samples_1456  samples_1711  samples_1945  samples_2180  samples_2405

# MT-Bench scores for those checkpoints, from the training log
samples_1456  6.21
samples_1711  6.55
samples_1945  6.81   <- best, selected
samples_2180  6.62
samples_2405  6.48

Score climbs, peaks at samples_1945, then falls. Later checkpoints saw more skills data and started to overfit it, trading general answer quality for a fractionally better fit to the training rows. This is ordinary model selection under a different name, the same overfitting curve the Data Science series draws under model evaluation and data leakage, and it is why a held out benchmark, not the epoch number, decides your model.

MT-Bench asks a stronger judge model, prometheus-8x7b-v2-0 here, to score multi turn answers on a ten point scale, so a checkpoint at 6.81 against another at 6.48 is not noise; it is a third of a point of judged answer quality across a benchmark of varied prompts. Small on paper, visible in a support chat where a wrong session timeout or a hedged non answer is the difference between a closed ticket and an escalation.

Checkpoints are not free to keep. Each phase two checkpoint is a full 14 GB copy of the model, so a run that saves ten of them writes 140 GB into the phase directory, and two or three runs will fill a modest data disk and make the next generation or training job fail with no space left on device. After you have recorded the best path, archive that one checkpoint and delete the rest; keeping every checkpoint from every run is how a tuning node quietly runs out of disk in the middle of a job.

Stopping the inference server before you train

My first attempt failed in under a minute. The assistant from Part 8 was still being served on the same node, holding the cards, and training could not allocate. RHEL AI general availability does not support training and inference serving at the same time, and the failure looks like a memory error even though the real cause is a busy GPU.

$ ilab model train --strategy lab-multiphase --phased-phase1-data ... --phased-phase2-data ...
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB.
GPU 0 has a total capacity of 44.35 GiB of which 96.00 MiB is free.

# the vLLM serve process from Part 8 was still holding all four cards
$ nvidia-smi --query-gpu=memory.used --format=csv,noheader
43722 MiB
43698 MiB
43705 MiB
43711 MiB

# stop serving, then retrain
$ pkill -f 'ilab model serve'
$ ilab model train --strategy lab-multiphase --phased-phase1-data ... --phased-phase2-data ...
Training Phase 1/2...

Full fine tuning is memory hungry for a reason worth internalising. Granite-7b-starter is about 14 GB at bf16, but training also holds gradients and AdamW optimizer state, which together push working memory well past 100 GB, so a 7B model that serves happily on one L40S needs the four card node to train. Serving needs weights plus a KV cache; training needs weights times roughly six. Different budgets, and the reason generation in Part 6 and training both live on the big node.

Here is the arithmetic that catches people. Weights are about 14 GB at bf16. Gradients match the weights at another 14 GB. AdamW keeps two optimizer moments per parameter in fp32, which for a 7B model is roughly 56 GB. Add activations and framework overhead and the working set clears 90 GB before any long context, which is why four 48 GB cards, sharded with tensor parallelism, are the floor for full fine tuning a 7B model and one card is not close. If that budget will not fit your hardware, the lever is a parameter efficient method such as LoRA, which trains a small adapter instead of the full weights, at some cost in quality.

Restarting a failed run without wiping checkpoints

Long runs fail. A card freezes, the vLLM eval server refuses to start, or a config error trips phase two. When multi phase training starts, it writes a journalfile.yaml recording where it is, so a re-run can pick up rather than start over. The prompt that governs this is the one that cost me five hours, and its wording is a trap.

$ ilab model train --strategy lab-multiphase --phased-phase1-data ... --phased-phase2-data ...
Metadata (checkpoints, the training journal) may have been saved from a previous training run.
By default, training will resume from this metadata if it exists
Alternatively, the metadata can be cleared, and training can start from scratch
Would you like to START TRAINING FROM THE BEGINNING? n   # n resumes from your checkpoints

# typing y here clears the journal and every saved checkpoint, and starts over
Would you like to START TRAINING FROM THE BEGINNING? y   # y wipes hours of work

Read it slowly. The question is whether to start from the beginning, so n means resume and y means wipe, which is the opposite of the reflex to type y for yes, continue. After a mid run failure you almost always want n. I now keep a copy of the phase directory before any re-run, because a journalfile is cheap insurance and a five hour phase one is not. Once training gives you a best checkpoint, confirm it serves before trusting it.

Most training failures fall into a short list. Keep this lookup beside the run so a red traceback becomes a two minute fix rather than a re-run from scratch.

$ ilab model serve --model-path ~/.local/share/instructlab/phase/phase2/checkpoints/hf_format/samples_1945
INFO instructlab.model.backends.vllm: vLLM engine successfully started

$ ilab model chat --model ~/.local/share/instructlab/phase/phase2/checkpoints/hf_format/samples_1945
>>> What is the default session timeout in Acme Relay 4.2?
The default session timeout in Acme Relay 4.2 is 15 minutes, reduced from 30 minutes in earlier releases.

That answer came from the seed context in Part 9, now baked into the weights rather than retrieved. Evaluating whether it holds up across questions it was not seeded on, and scoring it honestly against the base model, is Part 11.

Serving the checkpoint straight after training is a smoke test, not an evaluation. It tells you the weights load, vLLM accepts them, and a seeded question returns a sane answer, which rules out a corrupt or mis pathed checkpoint before you invest in scoring. Ask three questions you seeded and one you did not; if the unseeded one collapses into nonsense, the run overfit and no MT-Bench polish will save it.

One number frames whether nine hours was worth it: the base granite-7b-starter MT-Bench score before you tuned. If the base scored 6.4 and your best checkpoint scores 6.81, tuning bought you real ground; if the best barely clears the base, the seeds and not the trainer are the problem, and more epochs will not rescue thin data. Capture the base score once so every future run has an honest yardstick, which is exactly the held out baseline discipline the next part builds on.

Recommended training settings for the support assistant

For the assistant I run lab-multiphase, never skills only, because the whole point is teaching product facts and skills only training skips the knowledge phase. Training goes on the four card node with the inference server stopped first, and I set max_seq_len to 10000 in config.yaml unless a leaf genuinely needs longer context, which cut a nine hour run to about six. I keep the best checkpoint path RHEL AI names and archive the numbered siblings rather than reaching for the last one. On a failed run, n resumes, and I copy the phase directory before touching the prompt.

One judgement call worth naming: whether to tune a given leaf at all. If a fact changes weekly, a changelog line or a price, tuning it in is a mistake, because retraining to update one number is absurd and the tuned answer drifts stale. Tune the stable shape of the domain, the vocabulary, the product model, the way a good answer reads, and leave the volatile facts to retrieval later in the series. That split keeps this nine hour run rare rather than routine.

References

• Red Hat Enterprise Linux AI 1.5, training a model
• Red Hat Enterprise Linux AI 1.5, evaluating the model
• InstructLab, creating new knowledge or skills and the training phases
• InstructLab core CLI on GitHub

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts