Dr. Pranay Jha

You run docker pull nvcr.io/nvidia/pytorch:24.12-py3 in your CI job at 2 a.m. and it fails with a 401. You check the credentials, they look fine. You check the token expiry, it has not expired. The real problem is that the API key was scoped to a personal namespace and your CI agent is authenticating against the org namespace. NGC’s org/team/key model has a few sharp edges that catch everyone eventually. This part is about understanding the full NGC content supply chain before those edges catch you in production.

What Actually Lives in the NGC Catalog

The NGC catalog (catalog.ngc.nvidia.com) is a curated set of GPU-optimized software. Four content types live there, and they behave differently when you try to consume them programmatically.

Containers

These are OCI images stored in the nvcr.io container registry. NVIDIA publishes frameworks (PyTorch, TensorFlow, JAX), inference runtimes (TensorRT-LLM, Triton), NIM microservice images, CUDA development images, and HPC application containers. The registry space follows the pattern nvcr.io/<org>/<image>:<tag>, or nvcr.io/<org>/<team>/<image>:<tag> for team-scoped content. NVIDIA’s own published images live under nvcr.io/nvidia/. NIM images live under nvcr.io/nim/.

Pretrained Models

Model artifacts are not OCI images. They are versioned file archives downloaded via the NGC CLI or the NGC API. A pretrained model might be a set of checkpoint files, ONNX weights, or TensorRT engine files. You cannot docker pull a model; you use ngc registry model download-version. This trips up engineers who assume NGC is just a container registry.

Helm Charts

Helm charts on NGC are stored as versioned artifacts, not in a standard Helm repo index. You fetch them with helm fetch pointed at a specific NGC chart URL, or pull them via the NGC CLI. The GPU Operator Helm chart itself is published to NGC, though NVIDIA also mirrors it to the standard helm.ngc.nvidia.com repository. For Argo CD or Flux, you typically reference the standard Helm repo endpoint rather than the artifact download path.

Resources

Resources is the catch-all bucket: Jupyter notebooks, deployment scripts, configuration templates, and SDK bundles. You download these via NGC CLI just like model artifacts.

The nvcr.io Registry and How Authentication Works

The NGC container registry endpoint is nvcr.io. Authentication uses a special convention: the username is always the literal string $oauthtoken, and the password is your NGC API key. This is not a typo or a shell variable. The string $oauthtoken signals to the registry that you are authenticating with an API key rather than a user/password pair.

API keys are generated at org.ngc.nvidia.com. Each key is scoped to a specific NGC account (personal or org-level). If your CI system uses a key scoped to a personal account and the image lives in an org namespace, the pull will fail with a 401 even though the key is valid. The fix is to generate a key at the org level and store it in your secret store, not in a developer’s personal account.

The Org/Team Hierarchy

NGC has a three-level namespace: org > team > content. An org is the top-level account, typically matching your company or department. Teams are sub-groups within an org; you push private images to a team space with paths like nvcr.io/<org>/<team>/<image>:<tag>. Every org has at least one default team, but teams must be created and users assigned before pushes can succeed.

The org name is not always obvious. After logging in to ngc.nvidia.com, your org slug appears in the top-right of the dashboard. It is usually an alphanumeric string that does not match your company display name. Put this slug in your runbook immediately; tracking it down at 2 a.m. is unpleasant.

Operational Artifact: Pull-and-Deploy Flow

Below is the complete sequence: authenticate, list available images, pull a container, download a model artifact, and fetch a Helm chart via the NGC CLI. All commands HTML-escaped; expected output and failure modes included.

Step 1: Authenticate to nvcr.io

# Username is the LITERAL string $oauthtoken, not a variable
echo "${NGC_API_KEY}" | docker login nvcr.io \
  --username '$oauthtoken' \
  --password-stdin

# Expected output:
# WARNING! Your password will be stored unencrypted in /root/.docker/config.json
# Login Succeeded

# FAILURE MODE: if you see "unauthorized: authentication required"
# the API key is scoped to the wrong namespace (personal vs org).
# Generate an org-level key at org.ngc.nvidia.com/account/api-keys

Step 2: List Available Images via NGC CLI

# Install NGC CLI first: https://org.ngc.nvidia.com/setup/installers/cli
# Configure with: ngc config set  (enter org slug, API key, output format)

ngc registry image list nvidia/pytorch --format_type=ascii

# Expected output (truncated):
# +----------------------------------+-------+---------+--------+
# | Name                             | Tag   | Size    | Updated|
# +----------------------------------+-------+---------+--------+
# | nvidia/pytorch                   | 25.03-py3 | 18.4 GB | ...  |
# | nvidia/pytorch                   | 24.12-py3 | 17.9 GB | ...  |
# +----------------------------------+-------+---------+--------+

# List NIM images
ngc registry image list nim/ --format_type=ascii

# FAILURE MODE: "Invalid API key" or empty output with no error
# means the NGC CLI config has a stale key or mismatched org slug.
# Run: ngc config set  and re-enter credentials.

Step 3: Pull a Container

docker pull nvcr.io/nvidia/pytorch:25.03-py3

# Pull a NIM image (requires NVAIE entitlement for supported use)
docker pull nvcr.io/nim/meta/llama-3.1-8b-instruct:1.8.0

# Expected: standard docker pull progress bars, then:
# Status: Downloaded newer image for nvcr.io/nvidia/pytorch:25.03-py3

# FAILURE MODE: 403 Forbidden on NIM images without NVAIE entitlement
# The image may be listable but the pull will be denied.
# Verify entitlement at: org.ngc.nvidia.com/subscriptions

Step 4: Download a Pretrained Model

# Models are not OCI images - use NGC CLI
ngc registry model download-version \
  nvidia/riva/speechtotext_en_us_citrinet_1024_gamma_0_25:deployable_v2.0.0 \
  --dest /mnt/models/

# Expected output:
# Downloading ...
# Transfer id: ...  Download status: Completed
# Downloaded 2.1 GB to /mnt/models/

# FAILURE MODE: if the model is NVAIE-entitled and your key is personal,
# you get: "You do not have access to this resource."
# Org-level key + active NVAIE subscription required.

Step 5: Fetch a Helm Chart

# GPU Operator Helm chart via standard Helm repo (preferred for GitOps)
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm fetch nvidia/gpu-operator --version 25.3.0 [VERIFY]

# Alternative: NGC CLI for charts stored only as NGC artifacts
ngc registry chart download-version \
  nvidia/tao/tao-toolkit-api:5.5.0 \
  --dest ./charts/

# FAILURE MODE: helm fetch from helm.ngc.nvidia.com may require
# a repo token for entitled charts. Add credentials:
# helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \
#   --username '$oauthtoken' --password "${NGC_API_KEY}"

NVIDIA AI Enterprise Entitlement: What Actually Changes

This is the most misunderstood part of the NGC story. NVIDIA AI Enterprise (NVAIE) is an annual per-GPU subscription. The entitlement changes two things, not one.

The Enterprise Catalog, which used to be a separate portal from the public NGC catalog, was merged into the unified NGC catalog. An NVAIE-entitled user logging in to catalog.ngc.nvidia.com sees the same interface as a community user, but the entitled items carry a lock icon and become accessible once the entitlement is linked to the org account.

The key operational point: you can often pull an entitled image even with a community key if the image was recently public, because NVIDIA does not always lock images immediately on entitlement. Do not rely on this. In a production deployment, verify that your org account shows an active subscription under Subscriptions in the NGC dashboard before you build your deployment around those images.

Public vs Supported: The Two Lanes

There is a practical distinction between what NVIDIA publishes and what NVIDIA supports. The diagram below captures this. Most teams discover the hard way that "it’s on NGC" does not mean "NVIDIA will fix it if it breaks."

NGC in CI Pipelines: What to Wire and What to Avoid

A CI pipeline that pulls directly from nvcr.io on every build is a reliability risk. NVIDIA’s registry has rate limits and occasional maintenance windows. Framework containers are large (12 to 20 GB). A cold pull in a pipeline adds minutes and creates a single external dependency on every build.

The Right Pattern: Pull Once, Mirror Internally

Mirror images to an internal registry (Harbor, GitLab Container Registry, JFrog Artifactory) on a scheduled job. Your CI pipeline pulls from the internal mirror. This decouples build speed and reliability from NGC availability, and is the first step toward the air-gapped posture covered in Part 15.

What to Mirror vs What to Pull Live

The NGC Private Registry: When You Need It

The NGC Private Registry is the hosted namespace within nvcr.io where you store your own content: custom training containers, fine-tuned model checkpoints, internal Helm charts. It is included for DGX system customers and NVAIE subscribers. It is NOT a replacement for an on-prem container registry. It is a collaboration hub for sharing GPU-optimized content across a team without standing up your own registry infrastructure.

The push path is docker push nvcr.io/<org>/<team>/<image>:<tag>. The team must be created first in the NGC portal, and the user pushing must be a member of that team. If the push fails with a 403 and you are certain the API key is correct, check team membership before anything else.

For organizations with strict data residency requirements, storing custom containers in NGC Private Registry may not be acceptable even if the data is encrypted in transit. In those cases, the NGC Private Registry is useful only for public NVIDIA content consumption; custom artifacts stay on-prem in Harbor or a similar self-hosted registry.

Verdict: NGC as Source of Truth vs Mirroring

Use NGC as your source of truth for what NVIDIA ships. Use an internal mirror as your runtime source of truth for what your clusters consume. These are different things and should be treated differently.

Treat NGC as authoritative for: the canonical image tag to pull for a given framework version, the supported version matrix for GPU Operator and driver, the Helm chart values schema, and the list of CVE-patched releases. Subscribe to NGC release notifications for the software you run. When NVIDIA releases a patched image, your sync job picks it up and your mirror updates.

Do NOT treat NGC as authoritative for: runtime availability (it is not a CDN with an SLA for your cluster), image retention (NVIDIA deprecates old tags; a tag you pulled six months ago may be gone), or model completeness (not all models are on NGC; Hugging Face and self-hosted model stores fill gaps).

When not to use NGC at all: if your security policy prohibits pulling from external registries, skip NGC entirely and work directly with NVIDIA’s enterprise team to get offline media. If you are running a model that is not NVAIE-entitled and you need CVE coverage, NGC does not provide it; evaluate whether a community framework container from NGC is appropriate for a workload with enterprise security requirements.

What to validate before going to production: confirm the org API key is org-scoped and stored in a rotation-capable secret store; confirm the image tag you plan to deploy is still present in NGC (pin it to a digest, not just a tag); confirm your NVAIE subscription is active and linked to the org account if you are running entitled software; confirm your internal mirror job ran successfully and the CVE scan passed before promoting an image to prod.

References

• NGC Catalog User Guide — NVIDIA GPU Cloud Documentation
• NGC Private Registry User Guide — NVIDIA GPU Cloud Documentation
• NGC CLI Registry Command Reference
• Pulling and Running NVIDIA AI Enterprise Containers
• NGC Catalog — catalog.ngc.nvidia.com

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts