The first time someone on your team tries to fine-tune a 7B model on the shiny new 24GB workstation card, the run dies in under a minute with CUDA out of memory. Their instinct is to ask for a bigger GPU. The right answer is usually that they picked the wrong method. Full fine-tuning is the most expensive option on the menu, and most teams never need it. The job that genuinely required two 80GB GPUs can, with LoRA, land on the card already in the machine. This part is about making that call deliberately, because the method you choose decides how much VRAM you provision, how big your checkpoints get, and what you end up storing on the Hub.

The choice that decides your GPU bill

Fine-tuning is, in infra terms, a memory and capacity problem before it is a data-science problem. The training step does not just hold the model weights. For every trainable parameter the optimizer also tracks a gradient and, with Adam, two running statistics. Those buffers live in higher precision than the weights. So the question is not how big the model is, it is how many parameters you are training and what the optimizer has to carry for each one. That single decision is the difference between one card and a small cluster.

Here is the rough rule of thumb for a 7B model. Treat these as planning numbers, not a benchmark, because activations and sequence length move the real figure.

The checkpoint column is the part infra people underrate. A full fine-tune produces a fresh copy of the entire model every time you save. Five experiments and you are storing five 14GB artifacts. LoRA saves only the adapter, which is the trained delta, so the same five experiments cost you a few hundred MB total and every adapter still points back to one shared base. That is the model equivalent of layered images sharing a base in a registry. If you read Part 7, you already know why those adapter files arrive as adapter_model.safetensors rather than pickle.

The Trainer, in infra terms

The Trainer is the job runner. TrainingArguments is its config file. If you have ever written a batch job spec with resource requests and a checkpoint policy, this is the same shape. You hand it an output directory, a batch size, a few knobs for precision and saving, and it owns the loop. The arguments that matter for capacity are the ones that trade memory for time.

Three knobs do most of the work. per_device_train_batch_size is how much you push through a GPU at once, and it is the first thing to lower when you hit an out-of-memory error. gradient_accumulation_steps lets you keep a large effective batch while using small micro-batches, so you fake a batch of 16 on a card that only holds 4. gradient_checkpointing=True trades compute for memory by recomputing activations on the backward pass instead of storing them, which is often what gets a borderline job to fit.

LoRA with PEFT: what gets trained

LoRA keeps the original weight matrix frozen and learns a small update beside it, written as the product of two thin matrices A and B. The rank r sets how thin they are. A linear layer that was, say, 4096 by 4096 gets a learnable path of 4096 by 8 and 8 by 4096 when r=8, which is a rounding error in parameter count. At inference the update can be folded back into the base, so there is no extra latency once merged.

In current transformers you do not need to wrap the model in a separate object. The PEFT integration lets you attach an adapter directly with add_adapter, then hand the model to the Trainer as usual. The base is frozen for you and only the adapter carries gradients. This needs peft >= 0.18.0.

pip install -U transformers datasets peft accelerate

from peft import LoraConfig, TaskType
from transformers import (AutoModelForCausalLM, AutoTokenizer,
                          Trainer, TrainingArguments)

model = AutoModelForCausalLM.from_pretrained('google/gemma-2-2b')
tok = AutoTokenizer.from_pretrained('google/gemma-2-2b')

lora = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=8,
    lora_alpha=32,
    lora_dropout=0.1,
)
model.add_adapter(lora, adapter_name='sql')

args = TrainingArguments(
    output_dir='./out',
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    gradient_checkpointing=True,
    learning_rate=2e-4,
    bf16=True,
    logging_steps=10,
    save_strategy='epoch',
)

trainer = Trainer(model=model, args=args, train_dataset=train_ds)
trainer.train()
model.save_pretrained('./sql-adapter')

Expected: a training log ticking through steps with a falling loss, and a ./sql-adapter directory holding adapter_model.safetensors plus adapter_config.json, weighing in at single-digit MB rather than gigabytes.

Failure mode: drop the add_adapter line and the same script becomes a full fine-tune, the optimizer allocates buffers for all 2B+ parameters, and a 24GB card dies with CUDA out of memory before the first step finishes. The fix is not a bigger GPU, it is the adapter.

QLoRA: when even the frozen base will not fit

LoRA shrinks the optimizer, but you still have to hold the frozen base in memory. For an 8B model in bf16 that is around 16GB just to load before any training state. QLoRA closes that gap by loading the frozen base in 4-bit precision with bitsandbytes, dropping the resident base to a few GB, while the LoRA adapter still trains in higher precision on top. The result is that a model which needed a data-center card to fine-tune now fits on a single workstation GPU.

import torch
from peft import LoraConfig, TaskType
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type='nf4',
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    'meta-llama/Llama-3.1-8B',
    quantization_config=bnb,
    device_map='auto',
)

model.add_adapter(LoraConfig(
    task_type=TaskType.CAUSAL_LM, r=16, lora_alpha=32))

Failure mode: bitsandbytes needs a CUDA build that matches your driver. On a CPU-only box or a mismatched CUDA toolkit it imports fine and then errors at model load. Validate the bitsandbytes install on the exact GPU image before you schedule the job, not after.

Which method should you pick

My recommendation: make LoRA the default and require a justification for anything else. It is recommended because it gives you almost all of the task-adaptation benefit at a fraction of the GPU and storage cost, and because the small adapters are trivial to version and roll back. It is not the right call when you are changing the model deeply, for example retraining for a new language or a very different domain, where the low-rank update genuinely cannot carry enough change. Before you commit, validate two things: that your target_modules cover the layers that matter for your task, and that your evaluation set shows the adapter moving the metric you care about and not just lowering training loss.

What breaks

The first row is the one that costs people days. For common architectures like Llama, Gemma, and Qwen, PEFT has sensible default targets such as the attention projection layers, so you can often leave target_modules unset. For anything unusual, or when results are flat, set it explicitly. The third row is the operational trap: an adapter is useless without the exact base it was trained against, so record the base repo and revision the way you would record a base image digest. When the natural next control is scanning and access policy on those artifacts, the registry discipline from the Harbor for Beginners guide carries straight over to model adapters.

What I’d run

Default to LoRA, reach for QLoRA when the base will not fit, and reserve full fine-tuning for the rare deep change on hardware you own. The reason this matters to you specifically is that the method choice is a provisioning decision wearing a data-science costume. Pick LoRA and a job that looked like a two-GPU server request becomes a single card with small, versionable artifacts. The concept of fine-tuning itself, what it does to a model and when it beats retrieval, lives in the Generative AI guide. The same LoRA and SFT ideas on the NVIDIA stack are in the NeMo customization part if your pipeline runs there. Next, we get into the part that the rule-of-thumb math glossed over: spreading a real training job across multiple GPUs without rewriting your code. Try a LoRA run on a small dataset this week and watch the adapter come out at a few MB. That number tells you everything about why this is the default.

References

Transformers: Parameter-efficient fine-tuning (PEFT integration)
Transformers: Fine-tuning with the Trainer
PEFT: LoRA and LoraConfig reference

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts