Key takeaways
watsonx.ai gives you three ways to adapt Granite: prompt tuning, LoRA fine tuning, and full fine tuning. Prompt tuning freezes the model and trains a small soft prompt, so it is cheap, fast, and forgiving on small data. LoRA trains a set of low rank adapters and gets closer to full fine tuning at a fraction of the cost. Full fine tuning rewrites the weights and is rarely the first thing you should reach for. Match the method to the size of your labeled dataset, not to how much you want the model to improve.
You fine tuned Granite, and it came back worse than the base model. I have watched that happen on more than one customer project, and the cause is almost always the same. The method did not match the data. Someone had two hundred labeled rows and ran a full fine tune, so the model memorized the noise and forgot how to generalize. The fix was not more data or a bigger model. It was prompt tuning, which is designed for exactly that small data case. This part is about picking the right tuning method on watsonx.ai before you spend a training budget finding out the hard way.
Two ways to change a model
Start with the distinction that everything else hangs on. Prompt engineering does not change the model at all. You write a better instruction, the weights stay exactly as they shipped, and the model does its best with the context you gave it. Tuning is different. Tuning uses a labeled dataset to adjust something so the model gets better at one narrow task without you writing a cleverer prompt each time. The question is what you adjust.
Fine tuning changes the model weights themselves. You feed in labeled examples, the training run computes how wrong each prediction was, and it nudges the parameters to reduce that error. Full fine tuning touches every weight, which is powerful and expensive and easy to overdo on small data. Prompt tuning leaves every weight frozen. Instead it trains a short vector, a soft prompt, that gets prepended to your input so the frozen model reads your task the way you want. LoRA sits between the two: it freezes the base model but trains a small set of extra parameters, the low rank adapters, that get added back at inference time.
watsonx.ai wraps all three in the Tuning Studio, a managed experience inside your project, and in the ibm-watsonx-ai Python SDK. You do not stand up GPUs or write a training loop. You point at a dataset, pick a method and a base model, set a handful of parameters, and the platform runs the experiment and registers the result as a deployable asset. The skill that matters is choosing correctly, so the rest of this part is about that choice.
How prompt tuning actually works
Prompt tuning is parameter efficient in the strictest sense. The base model is frozen. What you train is a small block of numbers, initialized either from random values or from a text instruction you supply, that lives in front of every input as a soft prompt. Soft means it is not words. It is a set of AI generated vectors in the same space as the model embeddings, tuned by gradient descent to steer the frozen model toward your task. A human cannot read them, and that is fine, because the model can.
Because you are training a few thousand values instead of eight billion, the run is quick and the storage is tiny. You can tune several soft prompts for several tasks and swap them against one deployed base model. The trade is a ceiling. Prompt tuning cannot teach the model genuinely new knowledge or a new output format it has never seen. It biases what is already there. For classification, extraction, and routing, that is usually enough. The IBM tutorial I lean on tuned a Granite model to classify customer reviews and lifted accuracy from 93.1 percent on the base model to 98.3 percent after tuning, about five points, without touching a single base weight.
When does fine tuning beat prompt tuning?
Reach past prompt tuning when the task needs the model to learn something, not just to be steered. A new output schema it keeps breaking, a domain vocabulary it has never seen, a style it cannot hold across long outputs: these want fine tuning, because they need the weights to move. The catch is data. Fine tuning rewards larger, cleaner datasets and punishes small ones by overfitting. The rough rule I use is that a few hundred good examples suit prompt tuning, a few thousand justify LoRA, and you need real volume and a real reason before full fine tuning earns its cost.
| Method | What changes | Typical data | Best when |
|---|---|---|---|
| Prompt tuning | A frozen model plus a trained soft prompt | Hundreds of rows | Classification, extraction, routing on small data |
| LoRA fine tuning | Small low rank adapters, base weights frozen | Thousands of rows | New format or domain, near full quality, lower cost |
| Full fine tuning | Every weight in the model | Tens of thousands and up | Deep specialization with volume and a clear payoff |
The parameter counts behind that table are the reason each method costs what it does. Full fine tuning updates every weight, so call it one hundred percent of parameters in play. LoRA updates a small fraction, on the order of one percent, by training only the adapters. Prompt tuning and the related prefix tuning move even less, roughly a tenth of a percent, since the frozen model is untouched and only the prompt vectors learn. The chart puts those three on a log scale so the gap is visible rather than a sliver.
LoRA and the low rank trick
LoRA, low rank adaptation, is the method I reach for most on watsonx when prompt tuning hits its ceiling. The idea rests on a piece of linear algebra. Each layer of a model holds a large matrix of weights, and rank decomposition lets you approximate the change to that matrix with two much smaller matrices that multiply back to the original size. Those small matrices are the adapters. During tuning only they learn, the base model stays frozen, and at inference the adapter weights are added back to the base weights to produce output tuned for your task.
The practical payoff is that you get much of the quality of full fine tuning while training a tiny fraction of the parameters, so the run is faster and cheaper and the artifact is small. QLoRA goes further by quantizing the base model during tuning to shrink the memory footprint again. On watsonx.ai, starting with the 2.1.1 software release, you run LoRA and QLoRA experiments programmatically through the SDK rather than from the Tuning Studio interface. One constraint to note now so it does not surprise you later: LoRA in watsonx.ai works on non quantized base models, and when you deploy the adapter you deploy it into the same deployment space where the base model lives.
flowchart TD
A[Need better task results] --> B{Is a better prompt enough?}
B -->|yes| C[Ship the prompt]
B -->|no| D{How much labeled data?}
D -->|hundreds| E[Prompt tuning]
D -->|thousands| F[LoRA fine tuning]
F --> G{Quality still short?}
G -->|no| H[Deploy the adapter]
G -->|yes| I[Full fine tuning]
Tune a Granite model in Python
Here is a prompt tuning run against Granite through the ibm-watsonx-ai SDK. It sets a classification task, points at a data asset you uploaded earlier, and runs in the foreground so you watch it finish. Keep the parameters visible in the call, because these are the knobs you will actually turn.
from ibm_watsonx_ai.experiment import TuneExperiment
from ibm_watsonx_ai.helpers import DataConnection
experiment = TuneExperiment(credentials, project_id=project_id)
tuner = experiment.prompt_tuner(
name='ticket triage tuning',
task_id=experiment.Tasks.CLASSIFICATION,
base_model='ibm/granite-3-8b-instruct', tunable ids shift per release
tuning_type=experiment.PromptTuningTypes.PT,
num_epochs=12,
learning_rate=0.001,
batch_size=8,
accumulate_steps=16,
max_input_tokens=128,
max_output_tokens=2,
init_method='text',
init_text='Classify the ticket as billing, outage, or other. Ticket:',
verbalizer='classify {billing, outage, other} {{input}}',
auto_update_model=True,
)
details = tuner.run(
training_data_references=[DataConnection(data_asset_id=asset_id)],
background_mode=False,
)
print(tuner.get_run_status()) # -> completedExpected output is completed. After that you call tuner.get_model_id() and deploy it like any registered model. Failure modes to plan for: if the base_model id is not in the tunable list for your instance, run() raises before it does any work, so call client.foundation_models.PromptTunableModels.show() first and use an id it returns. If your dataset columns do not match the verbalizer pattern, the run starts and then the loss refuses to fall, which looks like a bad model but is really a data shape bug. I marked the base model id as a point to verify because the tunable model list changes across releases, and a current instance may expose newer Granite ids than the one shown here.
Worked example
Take the review classification task from the IBM tutorial. The base Granite model scored 93.1 percent accuracy. After a prompt tuning run with these parameters on a few hundred labeled reviews, the tuned model scored 98.3 percent, the gain plotted in Figure 1. The training touched no base weights and produced a soft prompt small enough to store beside the base model. Swap the task from reviews to tickets and the shape is identical: label a few hundred rows, tune, deploy, measure against the base model, keep it only if the gain is real on held out data.
Parameters that move the needle
Most tuning failures are not exotic. They come from a couple of parameters set carelessly. Epochs that are too high overfit a small dataset. A learning rate that is too high makes the loss bounce instead of settle. For LoRA, the rank sets how much capacity the adapters have, and a rank set too high on thin data wastes compute and invites overfitting. The table lists the ones worth understanding before your first run.
| Parameter | Method | Typical start | What it controls |
|---|---|---|---|
| num_epochs | Both | 10 to 20 | Passes over the data, too many overfits |
| learning_rate | Both | 0.001 range | Step size, too high and loss bounces |
| batch_size | Both | 8 | Examples per step, bounded by memory |
| accumulate_steps | Both | 16 | Simulates a larger batch without the memory |
| rank | LoRA | 8 to 16 | Adapter capacity, higher costs more |
| alpha | LoRA | Scales with rank | How strongly adapters affect the base |
| target_modules | LoRA | Attention layers | Which layers get adapters |
Read the learning curve after every run before you trust a number. watsonx gives you the loss plot through tuner.plot_learning_curve(). A curve that slopes down and flattens near a low value is a healthy run. A curve that jitters sideways means the learning rate is too high or the data is inconsistent. A curve that drops then climbs on validation is overfitting, and the fix is fewer epochs or more data, not a bigger model.
What a tuning run costs and where it bites
Tuning is billed the same way the rest of watsonx.ai is, through capacity unit hours against your plan, which I broke down in Part 5 on pricing. The cost of a run scales with how much you train, which is why the parameter chart in Figure 2 is also a cost chart. Prompt tuning runs are cheap and quick. LoRA runs cost more because they touch more parameters and usually more data. Full fine tuning is the expensive end and needs a payoff to justify it. The hidden cost is not the single run, it is the retuning: models get retired and replaced, and a soft prompt or adapter is tied to the base model it trained against.
When not to tune at all
Tuning is not always the answer, and knowing when to walk away from it saves the most money. If the problem is that the model does not know a fact, tuning is the wrong tool, because tuning shapes behavior rather than loading a knowledge base into the weights. That job belongs to retrieval, which I covered in the RAG parts of this series. If the model keeps citing things that are not in your documents, you have a grounding problem, and Part 10 on Granite Guardian is the check for that, not a tuning run.
Skip tuning too when a stronger base model or a better prompt closes the gap, because both are cheaper to build and far cheaper to maintain. A tuned model is a maintenance commitment: when the base model is retired, your soft prompt or adapter has to be retrained against its replacement, and that bill arrives on someone else’s schedule, not yours. Validate three things before you commit to a tuning run. Confirm the task is a behavior problem and not a knowledge problem. Confirm a plain prompt on the strongest available model does not already clear your bar. Confirm you have enough labeled, held out data to prove the tuned model is genuinely better and not just memorizing. If any of those fails, do not tune yet.
Start with prompt tuning, reach for LoRA only when it stalls
If you have a labeled dataset and a task Granite is close to but not quite nailing, run a prompt tuning experiment first. It is the cheapest way to find out whether tuning helps at all, and if a small soft prompt closes the gap you are done, at a fraction of the cost of anything heavier. Move to LoRA when prompt tuning plateaus below your bar and you have thousands of examples to feed it. Keep full fine tuning in reserve for the rare case with the data and the payoff to match.
For your first run this week, upload one clean labeled dataset, tune a soft prompt on a Granite model, and compare the tuned model against the base on data you held out. That single comparison tells you more than any rule of thumb. When you want to teach the model behavior no amount of tuning data can, the next step is instruction data at scale, which is where Part 12 on InstructLab picks up. If you want to see how another platform frames the same choice, my AWS Series part on Bedrock fine tuning covers the same ladder in Amazon terms.
References
- Prompt tune a Granite model in Python using watsonx, IBM Think
- Low rank adaptation (LoRA) fine tuning, IBM watsonx Documentation
- TuneExperiment, ibm-watsonx-ai Python SDK
- Introducing the IBM Granite 4.1 family of models, IBM Research
About The Author
DrJha