Adapter
State of B
Rank r
α (alpha)
Pick a scenario, or set the controls yourself on each tab.
What you are looking at: one weight matrix with a LoRA adapter, shrunk to 6 × 6 so you can read every number. The frozen matrix W does its normal job; the side path squeezes the input through A into just r numbers, expands it back with B, scales it by α/r and adds it on. Press Next to move a step or Play to run it.
Try this: run it with Before training selected and watch step 4: B is all zeros, so y comes out exactly equal to Wx. Then switch to After training and raise α: the same adapter now pushes y further from Wx.
What you are looking at: a real adapter for Llama 3.1. LoRA leaves the model's weights frozen and trains two thin matrices next to some of them. Only the thin matrices are trained and saved, so the result is an adapter of megabytes instead of a new model of gigabytes. Below it, the GPU memory each training method needs.
Try this: switch from Attention + MLP to q and v only and watch the adapter shrink about six times. Then switch to 70B and compare the LoRA and QLoRA bars: the frozen base, not the adapter, decides which GPU you need.
What you are looking at: one decode step of a server running one base model with many adapters. Every running request reads the shared base weights once, together; then each distinct adapter in the batch does a small multiply on only its own requests. Adapters beyond the GPU slot count (--max-loras) cannot join this step. The base model, rank and targets come from tab 2.
Try this: keep 32 requests and slide the distinct adapters from 1 to 16: the extra memory traffic grows with the number of adapters, not requests. Then drop the slots below the adapter count and see requests start to wait.
What you are looking at: the minimum number of 80 GB GPUs needed just to hold the weights of N fine-tuned versions, either as N merged full copies or as one base model with N adapters. The base model and adapter size come from tab 2.
Try this: pick 70B and slide from 1 to 100 versions. Merged copies grow by two GPUs per version; the adapter route stays at a handful, because every GPU serves every version's traffic.
Shapes from Meta's Llama 3.1 configurations: 8B has 32 layers, hidden size 4,096, MLP size 14,336 and key/value projections of 1,024; 70B has 80 layers, 8,192, 28,672 and 1,024. An adapter on a d × k matrix adds r × (d + k) parameters. LoRA starts A random and B at zero and scales the side path by α/r (Hu et al., 2021). Memory estimates are illustrative: full fine-tuning uses 16 bytes per parameter (16-bit weights and gradients plus 32-bit Adam state); LoRA keeps a 16-bit frozen base; QLoRA keeps a 4-bit base at about 0.55 bytes per parameter; adapters train at 16 bytes per adapter parameter; activations are a rough allowance of 4 GB for 8B and 12 GB for 70B with gradient checkpointing and modest batches. Decode step times are ideal memory-bandwidth times on H100-class GPUs (3.35 TB/s each); real kernels add overhead. Adapter load time uses PCIe 5.0 x16 at about 64 GB/s. Serving assumes 16-bit weights on 80 GB GPUs with 10% kept free, and ignores the KV cache, which you add per part 14. The 6 × 6 example uses made-up numbers.