RHEL AI is not something you install onto a Linux box. It ships as the whole operating system, delivered as one bootable image with the GPU driver, the container runtime, InstructLab and a vLLM based inference server already inside it. That single design choice explains almost every surprise in this install, so it is worth saying out loud before the first command runs.

Last part we chose the model and the card: granite-3.1-8b-instruct in a four bit build on a single L40S, with a rented multi GPU node reserved for tuning. This part stands that first box up, from a downloaded image to a login prompt to a running ilab that can pull the model. By the end you have a machine that answers to ilab and holds Granite locally; serving it to real traffic is Part 8.

Image mode and what it changes about installing

RHEL AI is built on image mode RHEL, which means the operating system is shipped and updated as a container image, so the root filesystem is immutable and versioned rather than something you edit in place. A tool called bootc manages that bootable image as your OS. Instead of a package set assembled on the machine, you boot a known image, and to change it you switch to a new image and reboot.

Here is where instinct fights the design. On any ordinary Red Hat box, a missing GPU driver means sudo dnf install and move on. Do that on RHEL AI and you will either be told the filesystem is read only, or your change will vanish at the next update, because the running root is not where the system lives. NVIDIA driver, CUDA, vLLM and InstructLab are already inside the image Red Hat ships. You do not add them, and you do not upgrade them one at a time. You move the whole box to a new image with bootc.

$ bootc status
No staged image present
Current booted image: registry.redhat.io/rhelai1/bootc-nvidia-rhel9:3.5
  Image version: 9.20260318.0 (2026-03-18)   # exact build string varies [VERIFY]
  Image digest: sha256:6f2c...
Rollback image: none

Read the booted image line. That tag, bootc-nvidia-rhel9:3.5, is the version of everything on the box at once, and the tag tracks the RHEL AI version (confirmed as :1.4 and :1.5 on those releases). When you later tune this model with InstructLab, which is one concrete answer to the fine tuning versus RAG versus prompting question, that reproducibility is the payoff for giving up in place edits.

Getting the ISO and sizing the disks

Download the RHEL AI ISO from the Red Hat customer portal, under downloads for Red Hat Enterprise Linux AI, and unzip it. You also want a Red Hat account and a Registry Service Account before you start, because you will need both to pull models later. Nothing about the download is unusual; the disk layout is where people get hurt.

Three numbers matter, and Red Hat states them as floors, not suggestions. InstructLab writes its data under /home by default, so /home wants at least 1 TB. The root path needs at least 120 GB, because a bootc update stages a second image before it switches, and that staging needs real headroom. Tuning later, which we do in Parts 9 and 10, wants about 3 TB of scratch. Plan the partitions once, up front.

I learned the 120 GB floor the expensive way. On the first assistant box I gave the root partition 60 GB, reasoning that image mode felt small and lean, and handed the rest of the disk to /home. It served fine for a month. Then a z-stream update landed, bootc upgrade tried to stage the new image, and it died halfway with No space left on device. The box stayed up on the old image, which felt lucky until I realised I could not update it at all without reprovisioning. That cost about half a day, and reprovisioning with the root at 150 GB fixed it for good.

Writing a Kickstart that will not wipe the wrong disk

You can install through the graphical installer, but for a repeatable box a Kickstart file is the honest choice. A Kickstart is a text file that answers every installer prompt in advance, so the machine provisions itself with no clicks. Below is a minimal one for RHEL AI 3.5 using the container image embedded in the ISO, with the partition sizes from the last section.

# rhelai-bootc.ks  (RHEL AI 3.5, embedded container image)
ostreecontainer --url=/run/install/repo/container --transport=oci --no-signature-verification

%post
# point future updates at the Red Hat registry; tag tracks the RHEL AI version
bootc switch --mutate-in-place --transport registry registry.redhat.io/rhelai1/bootc-nvidia-rhel9:3.5   # [VERIFY tag]
touch /etc/cloud/cloud-init.disabled
%end

network --bootproto=dhcp --device=link --activate

# name the target disk so you cannot wipe the wrong one
ignoredisk --only-use=nvme0n1
clearpart --all --initlabel --disklabel=gpt --drives=nvme0n1
reqpart --add-boot
part /     --fstype xfs --size=122880
part /home --fstype xfs --grow

firewall --disabled
services --enabled=sshd
user --name=cloud-user --groups=wheel --plaintext --password CHANGE_ME
sshkey --username cloud-user 'ssh-ed25519 AAAAC3Nza...'
rootpw --iscrypted locked
reboot

Embed that file into the ISO so the installer picks it up automatically. The mkksiso tool comes from the lorax package, and on a clean build host it is usually not installed, which is the first error most people hit:

$ mkksiso rhelai-bootc.ks rhel-ai-nvidia-3.5-x86_64-boot.iso rhelai-bootc-ks.iso
bash: mkksiso: command not found

$ sudo dnf install -y lorax
...
Complete!

$ mkksiso rhelai-bootc.ks rhel-ai-nvidia-3.5-x86_64-boot.iso rhelai-bootc-ks.iso
Adding kickstart rhelai-bootc.ks to ISO
Wrote rhelai-bootc-ks.iso

First boot and logging in

Boot the machine from rhelai-bootc-ks.iso. Because the Kickstart is embedded, installation begins on its own, partitions the drive, lays down the image, and reboots into image mode RHEL. When it comes back, log in over SSH with the credentials from the Kickstart. So far it feels like any RHEL server; the difference is everything already sitting inside the image.

Prove the AI tools installed by running ilab with no arguments. A usage banner means InstructLab is present and on the path.

$ ssh cloud-user@rhelai-01.internal
$ ilab
Usage: ilab [OPTIONS] COMMAND [ARGS]...

  CLI for interacting with InstructLab.
  If this is your first time running ilab, it is best to start with
  `ilab config init` to create the environment.

Commands:
  config    Command group for interacting with the config of InstructLab.
  data      Command group for interacting with the data generated by...
  model     Command group for interacting with the models in InstructLab.
  system    Command group for all system related command calls
  taxonomy  Command group for interacting with the taxonomy of InstructLab.

Aliases:
  chat      model chat
  generate  data generate
  serve     model serve
  train     model train

While you are here, confirm the accelerator is visible to the OS with nvidia-smi. If the driver in the image and your card disagree, you find out now rather than three commands later.

$ nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv
name, memory.total [MiB], driver_version
NVIDIA L40S, 46068 MiB, 570.86.15

Registry login and initializing ilab

Red Hat serves its indemnified Granite builds from registry.redhat.io, and you have to authenticate before you can pull them. This is the step that trips almost everyone on day one, because the natural move is to use your Red Hat website login, and that is not what the registry wants.

# first try, using the Red Hat SSO account: fails
$ podman login registry.redhat.io -u 'pranay@example.com'
Password:
Error: authenticating creds for registry.redhat.io:
  unable to retrieve auth token: invalid username/password

# create a Registry Service Account at access.redhat.com/terms-based-registry
# then use its generated username and token
$ podman login registry.redhat.io -u '1234567|support-assistant'
Password:
Login Succeeded!

Use a Registry Service Account, whose username looks like a numeric id and a name joined by a pipe, and paste its token when prompted for a password. Do not hardcode that token in a script or a Kickstart; let podman store it in the auth file, and read it from there. With the registry reachable, initialize InstructLab.

$ ilab config init
Generating config file and profiles:
  /home/cloud-user/.config/instructlab/config.yaml
Cloning taxonomy repository into /home/cloud-user/.local/share/instructlab/taxonomy...
Detected hardware: NVIDIA L40S
Selected training profile: nvidia/l40s-x4   # auto detected on 1.2 and newer
Initialization complete.

That command writes a config, clones the taxonomy repository we built in Part 5, and picks a training profile that matches the card. On older releases you chose the profile from a menu; from 1.2 onward the hardware is detected for you, though it is worth checking the choice, since a mismatched profile shows up much later as a failed training run. If this box is air gapped, opt out of Insights telemetry now with sudo mkdir -p /etc/ilab and sudo touch /etc/ilab/insights-opt-out; full disconnected setup is Part 29.

Downloading Granite and a first verification

Pull the starter model straight from the Red Hat registry. This is the indemnified granite-3.1-8b-instruct we settled on in Part 6, downloaded by repository reference rather than a pasted Hugging Face token.

$ ilab model download --repository docker://registry.redhat.io/rhelai1/granite-3-1-8b-instruct:3.5   # [VERIFY tag]
Downloading model from registry.redhat.io/rhelai1/granite-3-1-8b-instruct:3.5 ...
Pulling manifest, then 15.9 GB of weights...
Download complete.

$ ilab model list
+-----------------------------+---------------------+---------+
| Model Name                  | Last Modified       | Size    |
+-----------------------------+---------------------+---------+
| granite-3.1-8b-instruct     | 2026-07-28 09:12:44 | 15.9 GB |
+-----------------------------+---------------------+---------+

Before you move on, run one pass over the whole box. Keep this checklist; it is the fastest way to tell a healthy first boot from one that will fail an hour later during serving.

With the model on disk, the box is ready to serve. What that serving looks like, batch versus real time and the trade offs behind each, the Data Science series covers in general terms under serving machine learning models; Part 8 does it concretely with ilab and vLLM.

flowchart TD
  A[Download RHEL AI ISO] --> B[Write Kickstart with disk layout]
  B --> C[Embed Kickstart with mkksiso]
  C --> D[Boot ISO and auto install]
  D --> E[Reboot into image mode RHEL]
  E --> F[SSH in and run ilab]
  F --> G[podman login to registry]
  G --> H[ilab config init detects GPU]
  H --> I[ilab model download Granite]
  I --> J[Serve in Part 8]

Recommended first boot path for the support assistant

For the assistant box I would provision from an embedded image Kickstart, pin the install to a single named disk, give the root 120 GB and /home the rest up to at least 1 TB, and let ilab config init auto detect the L40S. Two mistakes are worth naming so you can avoid them: reaching for dnf install to fix a driver, which does not persist on an image mode system, and logging in to the registry with your Red Hat password instead of a Registry Service Account token, which simply will not work. On Monday, run bootc status and df -h /home on your box, confirm the booted image tag and a 1 TB /home, and only then download the model. Next in the series we serve granite-3.1-8b-instruct with ilab and vLLM and pull a first token out of it.

References

• Red Hat Enterprise Linux AI 3.5, Installing
• Installing RHEL AI on bare metal, Kickstart and bootc
• Red Hat Developer, configure RHEL AI and download a foundational model
• Red Hat Enterprise Linux AI 3.4 release notes, vLLM v0.18.0

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts