First sign the network was the problem, not my YAML: a KServe pod stuck on ImagePullBackOff with unauthorized: access to the requested resource is not authorized against registry.redhat.io, from a cluster that had no route to registry.redhat.io. Last part costed the assistant and found it earns its keep only because its data cannot leave the building. This part moves it into the environment that constraint implies, a disconnected data center where nothing can pull an image or a model from the internet, and shows what actually has to travel across the gap first.

Two disconnected postures, air gapped and proxied

Disconnected is not one thing, and the posture you are handed decides most of the work. A fully air gapped site has no route off the floor at all; content arrives on physical media through a one way transfer, and even DNS to a public name resolves nowhere. A proxied or restricted site keeps a narrow, audited path out, usually an allowlisted proxy that can reach a named set of registries and nothing else. Both are called disconnected in the ticket, and both break a default OpenShift AI install, but the air gapped case forces the sneakernet workflow while the proxied case can often mirror over the wire and skip the physical hop.

Get this classified before you touch a YAML file, because it changes your transfer plan, your storage budget and your change window. For the support assistant the mandate was the strict version: customer tickets and product data sit in a data center with no egress, so every image and every gigabyte of Granite weights has to be carried in and loaded once. That is the case worth teaching, since anyone who can do the air gapped path can trivially do the proxied one. One more distinction matters for planning: a proxied site can usually update in place on a maintenance window, while a truly air gapped site updates only when someone physically carries a new mirror in, so its refresh cadence is a logistics decision, not a scheduling one, and it tends to lag unless you make it a standing job.

Mirroring images with oc-mirror v2

Every container the cluster runs has to exist inside your boundary before the install starts. Red Hat ships oc-mirror for this, and version 2 replaced the version 1 workflow; v1 is deprecated, so build new work on v2. You define an ImageSetConfiguration listing the platform release, the operator catalogs you need and any extra images, mirror it to a directory on a connected host, carry that directory across, then push it into the internal registry. On the connected side:

# Tested against OpenShift 4.16, OpenShift AI (RHOAI) 2.16, oc-mirror v2,
# Red Hat AI Inference Server 3.2.1, RHEL AI 1.5, Granite 3.1 8B.
# Pull secret read from the standard podman auth path, never inlined:
#   $XDG_RUNTIME_DIR/containers/auth.json  (podman login writes it)

cat > imageset-config.yaml <<EOF
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
  platform:
    channels:
    - name: stable-4.16
      minVersion: 4.16.17
      maxVersion: 4.16.17
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.16
    packages:
    - name: rhods-operator          # OpenShift AI
    - name: gpu-operator-certified  # NVIDIA GPU Operator
    - name: nfd                      # Node Feature Discovery
  additionalImages:
  - name: registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.2.1-1756225581
EOF

# Mirror to a local workspace on the connected host:
oc-mirror --v2 -c imageset-config.yaml --workspace file:///data/mirror \
  docker://internal-registry.corp:8443/mirror

When it finishes it writes the objects the cluster needs, and this is the detail most older guides get wrong. oc-mirror v2 emits an ImageDigestMirrorSet and an ImageTagMirrorSet, not the ImageContentSourcePolicy you still see pasted around. ICSP is deprecated; apply the generated IDMS instead. Watch for the run to end on a warning rather than a clean exit, because a partial mirror looks a lot like success until a pod cannot pull three weeks later:

[INFO]   : Generating IDMS file...
[INFO]   : /data/mirror/working-dir/cluster-resources/idms-oc-mirror.yaml created
[INFO]   : Generating ITMS file...
[INFO]   : Generating CatalogSource file...
[INFO]   : mirror time     : 5h43m42s
[WARN]   : [Worker] some errors occurred during the mirroring.
 Please review .../logs/mirroring_errors_20260731.txt for a list of errors.

# The real failure hides in that log:
# error: unable to retrieve source image registry.redhat.io/rhaiis/vllm-cuda-rhel9
#   unauthorized: access to the requested resource is not authorized
#
# Cause: the podman auth.json used by oc-mirror had no registry.redhat.io
#   entry, only quay.io. Fix, on the connected host:
#   podman login registry.redhat.io     # writes the missing auth entry
# then re-run oc-mirror. A WARN exit is not a clean mirror; read the log.

On the disconnected side you push the workspace into the registry, then apply the generated cluster resources and turn off the default catalog sources, which point at the internet and will only error:

# Disable the catalogs that reach out to the internet:
oc patch OperatorHub cluster --type json \
  -p '[{"op":"add","path":"/spec/disableAllDefaultSources","value":true}]'

# Apply the digest mirror set and catalog source oc-mirror generated:
oc apply -f cluster-resources/
# imagedigestmirrorset.config.openshift.io/idms-oc-mirror created
# catalogsource.operators.coreos.com/redhat-operators created

oc get imagedigestmirrorset
# NAME              AGE
# idms-oc-mirror    22s

flowchart LR
  A[Connected host] --> B[oc-mirror v2 to disk]
  B --> C[Tarball on media]
  C --> D[Physical transfer]
  D --> E[Internal registry 8443]
  E --> F[IDMS redirects pulls]
  F --> G[Disconnected cluster]

Moving model weights across the gap

Here is the assumption that burns a day: oc-mirror does not carry your model. It mirrors container images, and the Granite weights are data on a volume, not a layer in the vLLM image. They cross the gap separately, and on the disconnected side they land on a PersistentVolumeClaim that the serving pod mounts at /mnt/models. A clean pattern is the throwaway copy pod: create the PVC, run a small UBI pod that mounts it, rsync the weights in, delete the pod. This mirrors the air gapped serving flow from the benchmarking work in Part 25.

# On a connected host, pull the weights once from the Red Hat registry.
# For RHEL AI the same weights come via ilab; here we stage files directly.
# 50Gi covers Granite 3.1 8B at bf16 with headroom.
oc create -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: granite-8b-weights
spec:
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 50Gi
EOF

# Temporary pod to mount the PVC (UBI, already mirrored):
oc create -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: model-copy-pod
spec:
  containers:
  - name: copy
    image: registry.access.redhat.com/ubi9/ubi:latest
    command: ["sleep", "3600"]
    volumeMounts:
    - name: models
      mountPath: /mnt/models
  volumes:
  - name: models
    persistentVolumeClaim:
      claimName: granite-8b-weights
EOF

oc rsync ./granite-3.1-8b/ model-copy-pod:/mnt/models/
oc delete pod model-copy-pod
# sent 15.98G  received 4.10K   ... 100%

Point the vLLM serving runtime at the PVC with --model=/mnt/models and it never reaches for Hugging Face or registry.redhat.io at run time. Size the transfer before you buy the drive: the numbers below are typical planning figures, not exact, and your operator subset will differ, but the shape is stable and the release image dominates.

Trusting your internal registry

A mirror registry on your own domain presents a certificate the cluster does not trust yet, and an untrusted registry fails the pull just as hard as a missing image, with a more confusing error. Grab the certificate, add it to the install config as an additionalTrustBundle, and on any RHEL host that also talks to the registry, install it into the system anchors. Do this before the install, not after the first failed pull:

# Pull the registry cert and trust it on a RHEL host:
openssl s_client -connect internal-registry.corp:8443 -showcerts </dev/null \
  | awk '/BEGIN/,/END/{print}' | sudo tee /etc/pki/ca-trust/source/anchors/reg.pem
sudo update-ca-trust

# In install-config.yaml, the same PEM goes under:
# additionalTrustBundle: |
#   -----BEGIN CERTIFICATE-----
#   ...
#   -----END CERTIFICATE-----

# Symptom when you skip this, from a node journal:
# x509: certificate signed by unknown authority
# Fix is the trust bundle above, not a registry restart.

Pull secrets and where they leak

A disconnected build is a security control, so do not undermine it with how you handle the credential that pulls the images. Read the pull secret from the podman auth path or a mounted Kubernetes secret; never paste it into a manifest, a shell history or, worst of all, a code block in a runbook. In the cluster the credential lives in the pull-secret secret in openshift-config, and the model download credential belongs in a Secret mounted into the copy pod, not in the pod spec. Rotating a leaked registry key inside an air gapped site is a physical visit, so the cost of a leak here is higher than on a normal cluster, not lower. The upstream guardrail discipline from the guardrails explainer applies to secrets as much as to model output.

Locking down egress and pod privilege

Air gapping the site is a physical control; it does not make the workloads inside it well behaved. A model server that has no reason to reach the internet should be told so explicitly, because a compromised container in a regulated site is exactly the scenario the air gap exists to contain. Apply a default deny egress NetworkPolicy to the serving namespace, then allow only the internal registry and the endpoints the assistant genuinely needs, such as the vector store from Part 27. That converts the air gap from an accident of the network into an enforced property of the workload, and it catches the case where someone later opens a proxy hole and forgets.

Privilege is the second lever. OpenShift runs pods under a Security Context Constraint, and the vLLM serving pods do not need a privileged one; the restricted-v2 SCC, which blocks running as root and drops most Linux capabilities, is enough for a GPU inference workload once the NVIDIA device plugin is in place. Reserve any elevated SCC for the GPU Operator components that actually require it, and audit for pods that quietly requested more. A disconnected build that still runs its model server as root has given up half the containment it paid for with the sneakernet.

RHEL AI on a disconnected server

The single server RHEL AI box has a lighter version of the same problem, plus one setting people miss. RHEL AI phones home to Red Hat Insights by default, which a disconnected host cannot reach, so opt out explicitly rather than letting it retry and log noise. Then log in to the registry once on a connected staging host, pull the Granite weights with ilab, and carry the model directory to the disconnected box. Downloading the weights is where the pull secret bites, and it is the one step people run last and least carefully:

# On the disconnected RHEL AI host, opt out of Insights:
sudo mkdir -p /etc/ilab
sudo touch /etc/ilab/insights-opt-out

# On a CONNECTED staging host, authenticate then download Granite for 1.5:
podman login registry.redhat.io          # reads user + key, writes auth.json
ilab model download \
  --repository docker://registry.redhat.io/rhelai1/granite-3-1-8b-instruct:1.5 \
  --release 1.5

# Common failure when the login step was skipped or the key expired:
# ilab model download ...
#   Error: unauthorized: access to the requested resource is not authorized
# Cause: no valid registry.redhat.io credential in auth.json.
# Fix: re-run podman login registry.redhat.io with a current service key,
#   confirm with: podman login --get-login registry.redhat.io
# then re-run the download. Never hardcode the key in the command.

Carry the downloaded directory to the disconnected server, drop it under the ilab models path, and serve it exactly as Part 8 did; nothing about serving changes once the weights are local. Treating the mirror refresh as a scheduled job rather than a one off is the same automation instinct the Data Science Series applies to CI/CD for machine learning pipelines, and it is what keeps a disconnected site from drifting a year behind on security fixes.

Run a mirror dry run before you disconnect

Most disconnected installs fail on something that was cheap to catch while the network was still up. Verdict: run oc-mirror to a file on a connected host first, read the generated IDMS and the real byte count, and use that to size your transfer and confirm every operator resolved, all before anyone schedules the change window. The pattern to avoid is treating the model like just another image and discovering at serve time that it never crossed the gap. Prefer the strict air gapped drill even for a proxied site, because a workflow that assumes no egress never surprises you when the proxy rule changes under you.

With the assistant now running where nothing can phone home, the series has one question left: how the whole Red Hat AI stack compares to the managed clouds, and what to learn next. Next part is the verdict.

References

• Red Hat OpenShift AI Self-Managed, installing in a disconnected environment
• Red Hat Developer, how oc-mirror version 2 enables disconnected installations
• Red Hat Developer, benchmarking with GuideLLM in air-gapped OpenShift clusters
• Red Hat Enterprise Linux AI 1.5, downloading large language models

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts