Dr. Pranay Jha

A team I work with spent two weeks configuring Triton for a single LLaMA-3 70B deployment before someone pointed out that a NIM container would have served the same model in under an hour. Another team did the opposite: they tried to wedge a computer-vision pre-processing step, an ONNX classification model, and a post-processing Python script into a single NIM and hit a wall immediately. NIM is not built for that. Triton is. The two tools solve different problems, and mixing them up costs real engineering time.

What Triton Inference Server Actually Does

Triton is an open-source inference server from NVIDIA that treats your file system as a model repository. You point it at a directory tree, each sub-directory is a model, each model declares its backend in a config.pbtxt file, and Triton loads them all at startup. A single Triton instance can simultaneously serve a TensorRT engine, a PyTorch TorchScript model, an ONNX model, a Python-based custom inference script, a vLLM-backed LLM, and a FIL tree-ensemble, all concurrently on the same GPU or across multiple GPUs.

The officially supported backends as of the current Triton release include: tensorrt (TensorRT engines), pytorch (TorchScript), onnxruntime (ONNX), tensorflow, python (arbitrary Python code via the Python backend), dali (GPU-accelerated pre-processing), fil (XGBoost, LightGBM, Scikit-Learn random forests via RAPIDS FIL), vllm (LLM serving), and tensorrtllm (TensorRT-LLM). Each backend is a shared library Triton loads dynamically. You can write your own backend in C++ against the Triton backend API.

The Model Repository Layout

Every model in Triton lives in a directory with a specific structure. The version sub-directory (1/, 2/, 3/ …) lets you run multiple model versions simultaneously. Triton version policy controls which versions get loaded: latest, all, or a specific list. This is live model-version management without restarting the server.

/models/
  resnet50/
    config.pbtxt
    1/
      model.plan
  text_detector/
    config.pbtxt
    1/
      model.onnx
  postprocessor/
    config.pbtxt
    1/
      model.py
  pipeline/
    config.pbtxt

Dynamic Batching

Dynamic batching is configured per-model in config.pbtxt. Triton queues incoming single requests and groups them into batches up to max_batch_size within a max_queue_delay_microseconds window before dispatching to the backend. The result: higher GPU utilization without the client needing to batch requests explicitly. Ensemble models route data through composing models and pass the dynamic-batching benefit down: the ensemble itself has no batching overhead since it is a pure event-driven router.

What NIM Actually Does

NIM is an opinionated, pre-packaged container that wraps one model with a curated inference runtime and an OpenAI-compatible REST API. When you start a NIM container, it inspects your GPU hardware profile, downloads the best matching model artifact from NGC (a TensorRT-LLM engine for supported GPUs, or a base weights set for others), and starts serving with zero configuration required.

NIM LLM v2.x introduced the one-container, one-backend architecture: the orchestration layer (nim-llm) handles licensing and hardware-aware profile selection; nimlib selects the optimal artifact; vLLM provides the actual inference engine and the OpenAI API endpoint. For select GPU/model combinations, TensorRT-LLM engines are downloaded and used instead of the raw weights. NVIDIA contributes optimizations upstream to vLLM so the distinction between the two paths is less pronounced than it was in 2024.

NIM is not a generic inference platform. It does one model, with one backend, serving OpenAI-compatible endpoints. That is its value: the decision surface is near-zero for the user.

Triton vs NIM: The Comparison Matrix

The Ensemble Pipeline: Where Triton Has No Peer

The pattern that comes up constantly in production computer-vision and NLP pipelines is: raw input arrives, needs pre-processing, gets routed to one or more models, and post-processing generates the final output. Triton handles this natively with ensemble models. The ensemble config.pbtxt describes a DAG of tensor flows between named models. The ensemble scheduler routes data without copying tensors in CPU memory: it uses shared memory and direct GPU buffer references where possible.

A typical RAG pipeline on Triton: a DALI step for image decoding, an ONNX embedding model, a Python step that queries a vector store, and a vLLM-backed LLM for generation. The entire pipeline declared in config.pbtxt and appears to the client as a single endpoint.

name: "vision_pipeline"
backend: "ensemble"
max_batch_size: 32

input  [{ name: "raw_jpeg"  data_type: TYPE_UINT8  dims: [-1] }]
output [{ name: "label"    data_type: TYPE_STRING dims: [1] }]

ensemble_scheduling {
  step [
    {
      model_name: "dali_preprocessor"
      model_version: -1
      input_map  { key: "raw_jpeg"     value: "raw_jpeg" }
      output_map { key: "preprocessed" value: "image_tensor" }
    },
    {
      model_name: "resnet50_trt"
      model_version: -1
      input_map  { key: "image_tensor" value: "image_tensor" }
      output_map { key: "class_logits" value: "class_logits" }
    },
    {
      model_name: "label_mapper"
      model_version: -1
      input_map  { key: "class_logits" value: "class_logits" }
      output_map { key: "label"         value: "label" }
    }
  ]
}

I triton.cc] Started HTTPService at 0.0.0.0:8000
I triton.cc] Started GRPCInferenceService at 0.0.0.0:8001
I triton.cc] Started Metrics Service at 0.0.0.0:8002
I model_repository_manager.cc] loading: dali_preprocessor:1
I model_repository_manager.cc] loading: resnet50_trt:1
I model_repository_manager.cc] loading: label_mapper:1
I model_repository_manager.cc] loading: vision_pipeline:1

Dynamic Batching Config in Detail

Production-grade config.pbtxt for a TensorRT classification model with dynamic batching and two concurrent instances pinned to separate GPUs:

name: "resnet50_trt"
backend: "tensorrt"
max_batch_size: 64

input  [{ name: "input"  data_type: TYPE_FP32  dims: [3, 224, 224] }]
output [{ name: "output" data_type: TYPE_FP32  dims: [1000] }]

dynamic_batching {
  preferred_batch_size: [ 8, 16, 32 ]
  max_queue_delay_microseconds: 5000
}

instance_group [
  { count: 1  kind: KIND_GPU  gpus: [0] },
  { count: 1  kind: KIND_GPU  gpus: [1] }
]

version_policy { latest { num_versions: 1 } }

max_queue_delay_microseconds: 5000 means Triton holds a request for up to 5ms waiting to build a preferred batch. Too high and you add latency; too low and you waste GPU cycles on tiny batches. The right value depends on your p95 target. The instance_group block pins one engine copy to GPU 0 and one to GPU 1, giving active-active serving across two physical GPUs from one model repository directory.

Health check after startup:

curl -s http://localhost:8000/v2/health/ready
# Expected: HTTP 200

curl -s http://localhost:8000/v2/models/resnet50_trt
# Expected: JSON with state READY

Where NIM Wins: Turnkey LLM Serving

For LLMs that appear in the NGC catalog with a curated NIM profile, NIM is the correct choice for most teams. The argument is not about performance; it is about the maintenance surface. With NIM you do not own the engine compilation, the batching scheduler tuning, or the OpenAI API implementation. NVIDIA owns all of that and ships it pre-validated. Your job reduces to setting NGC_API_KEY, picking the NIM container tag, and writing a Helm values file for Kubernetes resource requests.

For an application team building on top of Llama 3.1 70B Instruct, Meta Llama 3.3, Mistral, or NVIDIA Nemotron, NIM gives you a tested, NVAIE-supported container with a known SLA. That is worth more than theoretical maximum throughput you would need months to tune Triton toward.

NIM also composes with Triton. A Triton ensemble can call a NIM endpoint via the Python backend or via an HTTP step in Business Logic Scripting. You get Triton orchestration and NIM pre-tuned LLM serving in the same pipeline. This is common in RAG architectures: Triton handles embedding and retrieval steps, NIM handles the LLM generation step.

When NOT to Use Triton — and When NOT to Use NIM

When Triton Is the Wrong Tool

If your entire workload is one supported LLM and your team has zero background in Triton backends, config.pbtxt, or TensorRT engine compilation, do not start with Triton. The configuration learning curve is real. You will spend your first two weeks debugging backend mismatches and config.pbtxt syntax before you serve a single inference request. Use NIM, prove the model works, then evaluate whether Triton flexibility is worth adding.

Triton is also the wrong tool if your model is not in a framework Triton supports. Models with custom attention kernels tied to a specific library release, or models that require hardware-specific APIs outside the Triton backend list, do not fit cleanly into Triton backend model. For those you either write a custom backend (non-trivial) or use a different serving framework.

When NIM Is the Wrong Tool

NIM is the wrong tool for any of these scenarios: (1) Your model is not in the NGC NIM catalog. (2) You need to serve multiple models behind a single endpoint or implement preprocessing logic before the model call. (3) You need FIL-backed tree models, ONNX sklearn models, or DALI preprocessing as part of your pipeline. (4) You are doing active research where the model architecture changes weekly and you need to swap backends without repackaging a container. (5) You need direct access to vLLM or TRT-LLM configuration parameters that NIM does not expose as env vars.

Decision Tree: Triton or NIM?

Licensing and Operational Differences

Triton is open source under BSD-3. You pull it from NGC, run it, and owe nothing to NVIDIA beyond standard GPU driver licensing. The community edition is fully functional with no call limits or seat restrictions. If you want NVIDIA support for Triton in production, that comes via NVIDIA AI Enterprise, which also gives you a tested and supported Triton container image aligned with a specific NVAIE release.

NIM requires an NGC API key for the model pull. For development and limited evaluation, NVIDIA offers free API key access. Production on-prem deployment of NIM is covered under NVIDIA AI Enterprise licensing. The NIM Operator for Kubernetes handles the key injection and profile download lifecycle. Missing the API key at runtime produces a clear error: the container exits with NGC_API_KEY not set before the model loads.

See Part 16 on NIM microservices for the full NIM deployment lifecycle, and Part 17 on NIM autoscaling in production for the Kubernetes Helm patterns.

Workload-to-Tool Mapping

What to Validate Before Committing to Either

• NIM catalog check: Search build.nvidia.com and NGC for your model. If it is not there with a hardware-specific profile for your GPU SKU, NIM is not an option today.
• Triton backend availability: Pull your target Triton container image and run ls /opt/tritonserver/backends/. Confirm the backend you need is present and the version matches your model requirements.
• Engine compatibility: If using TensorRT via Triton, verify the TensorRT version in the container matches what you compiled your engine with. This is the single most common source of load failures.
• License availability: Confirm your NVIDIA AI Enterprise seat count covers the number of NIM containers you plan to run in production. Development API keys have rate limits [VERIFY exact limits].
• Latency budget: Triton dynamic batching adds up to max_queue_delay_microseconds of latency per request. If your SLA is sub-5ms p99, that 5ms delay budget is already consumed. Tune before committing.
• GPU memory budget: Running five Triton model instances concurrently uses GPU memory for all five simultaneously. Sum the footprint of each model version you intend to load. NIM simplifies this since it is one model per container with NVIDIA-published GPU memory requirements.

The Verdict

For LLMs that appear in the NGC NIM catalog on your GPU hardware: use NIM. The time-to-production advantage is not marginal; it is weeks. NVIDIA has already done the engine compilation, the scheduler tuning, and the API layer. You are not leaving performance on the table by not rolling your own Triton config; you are gaining reliability and NVIDIA support. The one exception is when you need vLLM or TRT-LLM parameters that NIM does not expose as env vars and that materially affect your workload behavior. In that case, run the vLLM or tensorrtllm backend inside Triton directly and accept the configuration overhead.

For everything else: Triton. Multiple models, ensemble pipelines, classic ML, custom Python logic, ONNX models, or any model not in the NIM catalog — Triton is the only NVIDIA-native answer. The configuration work is real but one-time per model. The payoff is a single, observable, version-controlled serving plane for your entire non-NIM model inventory.

When to run both: If your stack includes even one cataloged LLM and more than one non-LLM model, run NIM for the LLM and Triton for everything else. They compose cleanly: a Triton Python backend step can call a NIM endpoint over localhost HTTP. You are not choosing between them; you are assigning each workload to the tool it fits.

Next in the series: Part 20 covers NVIDIA Dynamo — disaggregated prefill/decode scheduling at cluster scale. Once you are running more than a handful of LLM replicas, Dynamo is where the conversation goes next. If you are mapping out your NVIDIA inference stack from scratch, start with the NVIDIA AI Complete Guide for the full 30-part series context.

References

• NVIDIA Triton Inference Server Documentation
• Triton Model Repository Guide
• Triton Ensemble Models
• Triton Batchers (Dynamic Batching)
• NVIDIA NIM for LLMs Overview
• Serving ML Model Pipelines with Triton Ensemble Models — NVIDIA Technical Blog
• NVIDIA Triton for Every AI Workload

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts