Here is the whole thing. No training, no fine tuning, no serving stack. Just a model running on your machine:

from transformers import pipeline

clf = pipeline(task='text-classification',
               model='distilbert/distilbert-base-uncased-finetuned-sst-2-english')
print(clf('This deployment finally went smoothly.'))
# [{'label': 'POSITIVE', 'score': 0.9998}]

Those three lines did far more than they look. The call resolved a repo id, pulled the model config, the tokenizer files and the weights from the Hub, cached them on local disk, instantiated the model in memory, moved it to a device, tokenized your text, ran a forward pass, and turned raw logits back into a label. The right way to read this as an infra person: it is a registry pull followed by a process that maps a large artifact into memory and runs it. Treat it like pulling and running a container image, not like calling a stateless HTTP API. The same questions apply. How big is the artifact, where is it cached, what does it need to run, and does it fit the box.

What pipeline() assembles

A pipeline is a thin wrapper around three stages: a preprocessor, the model, and a post-processor. For text that means a tokenizer turns your string into token ids, the model turns ids into logits, and a task specific post-processor turns logits into something you can use, a label and a score for classification, generated text for text generation. You can build all three by hand with AutoTokenizer and AutoModel, and later parts do exactly that when you need the control. The pipeline exists so your first run is one call instead of fifteen.

If you skip the model argument, the pipeline silently picks a default model for the task and pulls it. That is convenient for a demo and a problem in production, because you no longer know exactly which artifact entered your environment. Name the model every time. It is the same discipline as pinning an image tag instead of running :latest.

Your first run, and what lands on the box

Install the library with the PyTorch backend, plus Accelerate for device placement:

pip install 'transformers[torch]' accelerate
# pulls transformers, the torch backend, and Accelerate for device placement

Now run a small generation model on the first GPU, in half precision:

import torch
from transformers import pipeline

gen = pipeline(
    task='text-generation',
    model='google/gemma-2-2b',
    device=0,
    dtype=torch.bfloat16,
)
out = gen('The secret to a clean rollback is', max_new_tokens=20)
print(out[0]['generated_text'])
# The secret to a clean rollback is having a tested path back to the last known good state ...

Two failure modes show up here, and both are infra problems, not model problems. The first is a 401 or a gated repo error: google/gemma-2-2b requires accepting a license and an authenticated token, which is why Part 3 set that up. The second is an out of memory error on load. That is not a bug, it is capacity. A model needs roughly its parameter count times the bytes per parameter just to hold weights, before any activations or KV cache. The worked example below makes that concrete.

nvidia-smi --query-gpu=memory.used --format=csv
# memory.used [MiB]
# 5312 MiB

Where the weights go: device, device_map and dtype

Three parameters decide placement, and they are the ones worth memorizing. device takes a single target: -1 for CPU, which is the default, or a GPU ordinal like 0 for the first card. device_map='auto' hands placement to Accelerate, which fills the fastest device first and then spills to CPU and disk if the model does not fit. dtype sets the precision the weights load in, and in current transformers this is the parameter name to use. Half precision, torch.float16 or torch.bfloat16, halves the VRAM footprint versus float32 with negligible quality loss on most models.

The precision choice is the cheapest lever you have. Here is the same 2.6 billion parameter model under each common dtype, weight memory only:

These are weight estimates, not the full runtime footprint. Activations and, for generation, the KV cache add on top and grow with sequence length and batch size. Treat the weight number as a floor. The int8 and int4 rows come from quantization, which the GenAI series covers as a concept in the Generative AI guide. Part 13 of this series does it with optimum and the Hugging Face tooling.

Choosing the device setting

The decision is short. No GPU, use the CPU default and accept it is for prototyping, not throughput. One GPU and the model fits, target it directly and set a half precision dtype. Model too big for one card, hand it to Accelerate and consider quantization to claw the footprint back down.

For the model that does not fit, the call is barely different. You swap the single device target for the map:

gen = pipeline(
    task='text-generation',
    model='google/gemma-2-2b',
    device_map='auto',
    dtype=torch.bfloat16,
)
# Accelerate places layers across GPU, then CPU, then disk as needed

When to batch, and when to skip it

Pass many inputs as a list and add batch_size to send several through the model per forward pass. On a GPU with regular input lengths this raises throughput. It is off by default for a reason: it can also do nothing, or push you into an out of memory error, depending on hardware, sequence length and the model.

prompts = ['ticket one text', 'ticket two text', 'ticket three text', 'ticket four text']
for result in clf(prompts, batch_size=2):
    print(result)
# pipeline sends two inputs through the model at a time

Quantization in one extra argument

When half precision still does not fit, the pipeline accepts a quantized model through model_kwargs. With bitsandbytes installed, load in 8 bit and cut the weight footprint roughly in half again:

import torch
from transformers import pipeline, BitsAndBytesConfig

gen = pipeline(
    model='google/gemma-2-2b',
    dtype=torch.bfloat16,
    device_map='auto',
    model_kwargs={'quantization_config': BitsAndBytesConfig(load_in_8bit=True)},
)
print(gen('Summarize the incident in one line:', max_new_tokens=30)[0]['generated_text'])
# requires: pip install bitsandbytes

Eight bit loading trades a small amount of quality for a large amount of VRAM, and for most chat and classification work the difference is hard to see. Below 8 bit the trade off becomes real and model dependent, which is why quantization gets its own part later rather than a footnote here. The registry side of this, scanning and trusting what you pull before it runs, parallels exactly what a container registry does for images; the Harbor for Beginners guide covers that control plane.

What I’d ship

Use the pipeline for your first runs, for prototypes, for batch jobs, and as the reference implementation you compare a serving stack against. It is the right default because it is the shortest correct path: it handles tokenization and post-processing that are easy to get subtly wrong by hand, and the placement knobs, device, device_map and dtype, cover the cases that matter on a single host.

It is not the right tool for production serving under real traffic. The pipeline runs one process, holds one copy of the model, and does not do request batching, queueing, or autoscaling. For that you move to Text Generation Inference or Inference Endpoints, which Parts 11 and 12 cover, or to the NVIDIA serving stack in the NVIDIA AI guide when you own the GPUs. Before you ship anything, validate three things: run nvidia-smi after load and confirm the model fits with headroom rather than offloading, confirm the dtype you asked for is what loaded, and pin the model revision so the artifact cannot change under you.

Pull a model that fits your card, run those three lines, and keep nvidia-smi open in another pane while it loads. The number it prints is the only capacity figure you can trust.

Where the weights live after the first pull

The first pipeline call downloads weights once and caches them on local disk. Every later call for the same repo reads from that cache instead of hitting the network, the same way a container runtime reuses pulled layers. By default the cache sits under the home directory at ~/.cache/huggingface/hub, which is fine on a laptop and wrong on most servers, where home is small and often on the wrong volume. Point it at real storage with the HF_HOME environment variable before the first run.

export HF_HOME=/mnt/models/hf-cache
python run_inference.py
# weights now cache under /mnt/models/hf-cache/hub, not the home directory

This is a storage and data movement decision, not a detail. Model repos run from hundreds of megabytes to tens of gigabytes, the cache only grows, and on a fleet of identical workers each node pulling the same model independently wastes bandwidth and time. The fixes are familiar from any artifact system: put the cache on a sized, monitored volume, warm it ahead of a rollout, and on air gapped hosts mirror the repo internally so production never reaches for the public Hub. Part 6 takes this further for datasets, where the cache problem is larger and easier to get wrong.

References

• Transformers documentation: Pipeline tutorial
• Transformers documentation: Pipelines API reference
• Hugging Face Transformers on GitHub

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts