Here is the whole product in one line: docker run --gpus all -p 8080:80 ghcr.io/huggingface/text-generation-inference:3.3.5 --model-id .... That is it. TGI is not a framework you integrate or a library you import. It is a container you deploy, scale, and monitor, exactly like every other stateful service you already run, except this one wants a GPU and a fast disk. If you have ever run a database in a container with a mounted data volume and a health check, you already understand the shape of this. The model is the data. The GPU is the constraint. Everything else is operations you have done before.
What the container does for you
The reason to run TGI instead of writing a loop around transformers is everything it does between the network socket and the GPU. A request comes in, a router validates and tokenizes it, and then it joins a running batch on a model shard. That batching is continuous: TGI does not wait to assemble a fixed batch, it adds and removes requests from the in-flight batch every step, which is what keeps a GPU busy under mixed traffic. It streams tokens back as they are produced, splits one large model across several GPUs with tensor parallelism, exposes an OpenAI-compatible Messages API, and ships Prometheus metrics and OpenTelemetry tracing. None of that is code you want to own.
The concepts underneath (quantization, attention, the KV cache) are covered vendor-neutral in the GenAI series. Here the job is to operate them. Think of the router as your reverse proxy and admission control, the shards as worker replicas pinned to specific GPUs, and the cache volume as the data directory you must not lose on restart.
The launch command, and what it pulls
This deploys a 7B model on a single NVIDIA GPU. The -v volume is the cache: mount it so weights survive a container restart instead of re-downloading gigabytes every time.
model=teknium/OpenHermes-2.5-Mistral-7B
volume=$PWD/data
docker run --gpus all --shm-size 1g -p 8080:80
-v $volume:/data
ghcr.io/huggingface/text-generation-inference:3.3.5
--model-id $modelExpected: after the weights download and shards warm up, the log ends with a line reporting the router is Connected and listening. A request to http://127.0.0.1:8080/health returns 200.
Failure modes you will hit:
A 401 or 403 on a gated model means you did not pass a token; add -e HF_TOKEN=hf_... to the run. A CUDA out-of-memory during warmup means the model plus your token budget does not fit; quantize it or lower --max-total-tokens. A silent hang during load on a multi-GPU run almost always means --shm-size is too small. And if Docker reports it cannot find the GPU, the NVIDIA Container Toolkit is not installed or --gpus all is missing.
--shm-size 1g is not optional for sharded runs. TGI uses shared memory for inter-shard NCCL communication, and the Docker default of 64 MB causes cryptic hangs or crashes the moment you set --num-shard greater than 1. If a multi-GPU launch freezes during model load with no obvious error, check this first.The flags that decide whether it fits
Run docker run ghcr.io/huggingface/text-generation-inference:3.3.5 --help and you get dozens of options. These are the ones that change whether the model loads, fits, and survives load. Everything else has a sane default.
| Flag | What it controls | Why the platform owner cares |
|---|---|---|
--model-id | Which repo to pull and load | Your artifact reference. Pin --revision to a commit for reproducible rollouts. |
--num-shard | Tensor-parallel shards across GPUs | One model split over N GPUs. Wants fast interconnect (NVLink) or latency suffers. |
--quantize | awq, gptq, fp8, bitsandbytes-nf4, eetq | Cuts VRAM, costs some latency or accuracy. Fits a model on smaller cards. |
--max-input-tokens | Longest prompt accepted | Request validation ceiling and a driver of memory use. |
--max-total-tokens | Prompt plus generated, per request | The single most important sizing knob. It is your memory budget. |
--max-concurrent-requests | Requests held before refusing (default 128) | Backpressure. A clean refusal beats a melted GPU. |
--cuda-memory-fraction | Cap on visible GPU memory used | Leaves headroom or shares a card with another workload. |
Worked example
A 7B model in fp16 needs about 14 GB just for weights (2 bytes times 7 billion parameters). Load that on a 24 GB card (an L4 or A10-class GPU) and you have roughly 8 to 9 GB left after framework overhead for the KV cache. That remaining memory is what --max-total-tokens spends: every concurrent request reserves space for its prompt plus generated tokens. Set it to 4096 and a handful of long-context requests will exhaust the cache and queue. The fix is not a bigger number, it is a smaller model, a quantized one (--quantize awq drops weights to about 4 GB), or a bigger card. TGI infers a safe --max-batch-total-tokens for you, but still set --max-total-tokens to match the prompt sizes you serve.
Consuming it: /generate vs the Messages API
TGI exposes two ways in. The native /generate endpoint takes an inputs string and returns generated text. More useful for most teams is the Messages API at /v1/chat/completions, which is wire-compatible with the OpenAI Chat Completions API. That compatibility is the real unlock for a platform team: existing clients, SDKs, and gateways that already speak OpenAI point at your endpoint with only a base URL change.
curl http://127.0.0.1:8080/v1/chat/completions
-X POST
-H 'Content-Type: application/json'
-d '{"model":"tgi","messages":[{"role":"user","content":"Name three things to monitor on an LLM server."}],"max_tokens":80,"stream":false}'Expected response (trimmed): {"object":"chat.completion","choices":[{"message":{"role":"assistant","content":"Latency to first token, GPU memory, and request queue depth."}}]}. Set "stream":true and you get server-sent events token by token instead.
This is the same three-way decision from Part 11, where I compared the hosted Inference Providers and managed Endpoints against self-hosting. TGI is the engine the self-host column runs on, and it is also what powers many managed Endpoints under the hood.
--max-concurrent-requests so a spike returns a clean 429 instead of dragging the GPU to a crawl. Scrape the Prometheus port (default 9000) into the same dashboard as the rest of your fleet, and alert on queue depth and time-to-first-token, not just CPU. The weights are large and slow to pull, so mount a persistent cache volume or pre-seed it, or every pod restart re-downloads gigabytes and your rollout stalls. This is capacity and dependency management, the same discipline you already apply to any stateful service.Health, metrics, and scaling it like a real service
TGI gives you the hooks you already wire for everything else. /health is your readiness and liveness probe. The Prometheus metrics live on a separate port (default 9000, set with --prometheus-port) so you can scrape them without exposing them on your serving port. Add --otlp-endpoint to ship traces to your collector, and --json-output to get structured logs your aggregator can parse. For multi-GPU, --num-shard 4 splits one large model across four cards on a node.
docker run --gpus all --shm-size 1g -p 8080:80
-v $PWD/data:/data
ghcr.io/huggingface/text-generation-inference:3.3.5
--model-id meta-llama/Llama-3.1-70B-Instruct
--num-shard 4
--max-input-tokens 4096
--max-total-tokens 8192
--quantize fp8Scaling out, not just up
One thing to be clear about: --num-shard scales a single model across GPUs for capacity and speed, it does not give you high availability. For that you run multiple TGI replicas behind a load balancer, each holding its own copy, so a node failure drops capacity instead of the service. This is the parallel to a stateless web tier, except each replica is expensive and slow to start, which changes your autoscaling math. Scale on queue depth and latency, give new replicas a generous warmup before they take traffic, and keep a warm pool if your traffic is spiky. The container registry parallel is exact: the Hub is to model weights what a registry is to images, and pulling a 70B model is a far heavier operation than pulling an app image, which is its own argument for an internal mirror, covered in the Harbor series.
The elephant: TGI is in maintenance mode
You should know this before you build a platform around it. As of 2026 the TGI GitHub repository is archived and read-only, and the project is in maintenance mode. TGI was the toolkit that pushed optimized inference engines toward shared transformers model definitions, and Hugging Face now contributes to and recommends downstream engines going forward: vLLM and SGLang for servers, llama.cpp and MLX for local. There is an official migration path from TGI to vLLM. On Hugging Face Inference Endpoints, the serving engine is fixed for the life of an endpoint, so moving from TGI to vLLM means creating a new endpoint, not flipping a setting.
Maintenance mode does not mean broken. TGI still serves production traffic, the container still pulls, the Messages API still answers. But maintenance mode does mean you should not expect major new features or fast support for the newest model architectures. That changes the calculus for a brand-new build.
| Consideration | TGI | vLLM |
|---|---|---|
| Status in 2026 | Maintenance mode, repo archived | Active, the recommended path |
| Strength | Drop-in HF container, simplest start | Throughput, new models, active dev |
| Run shape | One docker run | Server with OpenAI-compatible API |
| On HF Endpoints | Available, engine fixed per endpoint | Available, new endpoint to switch |
| Choose when | You already run it or want HF-native simplicity | New build wanting the supported road |
--api-key or a gateway) in front of any endpoint reachable beyond localhost. TGI does not authenticate requests by default. Format and provenance checks on the weights themselves are covered in Part 7 on safetensors.The short version
Why I still reach for TGI: it is the fastest way to turn a Hugging Face model into a real, OpenAI-compatible endpoint, and the container does the hard parts (batching, streaming, sharding, metrics) with no glue code. For a quick internal service or a workload you already run on it, it is hard to beat. When I would not: a brand-new platform I expect to maintain for years, because the project is in maintenance mode and the supported road leads to vLLM and SGLang. What to validate first: that your target model is supported by the TGI version you pin, that it fits VRAM at your real --max-total-tokens, and that switching engines later is a new-endpoint operation on Inference Endpoints rather than a config flip. If you run on NVIDIA hardware and want the vendor-tuned path, the TensorRT-LLM backend and the broader stack are the next stop in the NVIDIA AI series. For on-prem and air-gapped serving, the Private AI series covers the deployment substrate.
Your move: pull a 7B model with the first command above, point one curl at /v1/chat/completions, and watch the Prometheus port. You will know within an hour whether TGI fits your box and your traffic.
References
TGI Quick Tour
TGI launcher arguments reference
TGI Messages API
TGI GitHub (archived, maintenance mode)
About The Author
DrJha