AI infrastructure · GPU clusters · Kubernetes · Fabric · Observability · Automation

Read ten job descriptions for AI infrastructure roles across different companies and the same six things come up every time: run large GPU clusters, keep the fabric healthy, schedule GPU work on Kubernetes or Slurm, see everything in Prometheus and Grafana, automate it with Terraform and Ansible, and lead the customer through all of it. Most of us have solutioned those things. Fewer of us have typed the commands. This workbook is 100 lab tasks, in the order you would learn them, so you can.

FirstWhat these roles actually test

I built the list from the job descriptions themselves rather than from a course outline. Grouped, the requirements come to six areas, and every section below maps to one of them:

SecondWhere to run the tasks

Every task is tagged with the cheapest environment that can run it. You do not need a GPU cluster, and you do not need a GPU for most of it.

Do the B tasks in one sitting with the instance running, in order, and capture the output of every command as you go. Those captures are what you talk about in the interview.

ThirdWhy the order matters

The sections are in dependency order, not in order of interest. Section B prepares a Linux host so the driver install in section C succeeds. Section C gives you nvidia-smi and DCGM so the schedulers in D and E have something to hand out. Section F needs the DCGM exporter from E. Section H needs the fabric understanding from G to read its own output. Section I automates everything B to H did by hand, which only makes sense once you have done it by hand. Sections L and M turn all of it into the documents and the spoken answers the role is paid for. Skip ahead if you must, but the prerequisites table on each task will send you back.

Workbook100 tasks, in learning order

Section A is on paper. B to K are hands-on, with the environment tag in grey at the right. L is the documents. M is eight scenarios against the clock, the way they are asked.

Draw and write before touching a machine. Everything later refers back to these pages.

1. 1Draw an 8-GPU HGX node from memory: GPUs, NVSwitch, two CPU sockets, PCIe switches, NICs and local NVMe. Mark which NIC serves which GPU.P›
2. 2Draw a rail-optimised InfiniBand fabric for 32 nodes: one leaf per rail, a spine layer, and the path two GPUs on different nodes take to each other.P›
3. 3Write the software stack of a GPU node from the bottom up: firmware, kernel, driver, CUDA, NCCL, container runtime, scheduler agent, exporters.P›
4. 4Write how an NCCL all-reduce uses NVLink inside a node and the fabric between nodes, and why one slow link caps the whole job.P›
5. 5Write an acceptance plan for a new GPU cluster, layer by layer, naming the tool and the pass criterion at each layer.P›
6. 6Write a one-line vocabulary sheet: Slurm, BCM, GPU Operator, Network Operator, UFM, DCGM, NCCL, GPUDirect RDMA, Kueue.P›

A plain VM is enough. Every item here is a prerequisite the driver install will fail without.

1. 7Build a RHEL 9 or Ubuntu 22.04 VM and apply the GPU host pre-checks: nouveau blacklisted, kernel headers present, IOMMU and hugepages set in GRUB.A›
2. 8Inspect NUMA and PCIe locality with lscpu, numactl -H and lstopo, and write which CPU socket each PCIe slot belongs to.A›
3. 9Harden SSH: keys only, no root login, sudo with logging. Prove it with a failed password attempt in the log.A›
4. 10Leave SELinux or AppArmor enforcing and show a confined service still runs. Record how you read a denial.A›
5. 11Run a CIS benchmark check with OpenSCAP and fix five findings without breaking the GPU prerequisites.A›
6. 12Pin the kernel and disable unattended kernel upgrades. Write why a surprise kernel breaks a GPU node.A›
7. 13Ship auditd and journal logs off the node with Grafana Alloy or rsyslog to a collector.A›
8. 14Write the DOCA-OFED install procedure for the kernel you pinned and keep it as a runbook.A›

One rented GPU instance for a few hours. Start from a bare image so you see every step.

1. 15From a bare image, install the NVIDIA driver and CUDA toolkit by hand. Verify with nvidia-smi and nvcc --version.B›
2. 16Install the NVIDIA container toolkit and run a CUDA container. Then run one with a mismatched CUDA image and read the error.B›
3. 17Read nvidia-smi -q end to end and record driver, VBIOS, ECC mode, persistence mode, clocks and power limits.B›
4. 18Run nvidia-smi topo -m and explain every cell of the matrix: NV#, PIX, PHB, SYS.B›
5. 19Enable MIG on an A100 or H100, create two instances, run a job on each, then disable MIG.B›
6. 20Enable persistence mode as a systemd service and set a power cap. Confirm both survive a reboot.B›
7. 21Install DCGM and run dcgmi discovery, dcgmi health -c, dcgmi diag -r 1 and then -r 2. Record the run times.B›
8. 22Read ECC and row-remapper state with nvidia-smi -q -d ECC,ROW_REMAPPER and write the drain rule for each state.B›
9. 23Grep dmesg for Xid lines, map the common Xids to actions from the NVIDIA catalogue, and write a one-page Xid card.B›

The HPC scheduler most GPU clusters run. Containers on the laptop first, then the real GPU.

1. 24Bring up slurm-docker-cluster with two compute nodes and confirm sinfo shows both idle.A›
2. 25Submit jobs with sbatch and srun; hold, release and cancel one; read the squeue reason for a pending job.A›
3. 26Drain a node with a reason, watch what happens to its running job, resume it, and read sinfo -R.A›
4. 27Create two partitions with different time limits and a QoS, and submit a job to each.A›
5. 28Declare fake GPUs in gres.conf, submit with --gres=gpu:2, then ask for more GPUs than exist.A›
6. 29Enable accounting and read sacct for a finished job: elapsed time, CPU, memory and exit code.A›
7. 30On the GPU node, install single-node Slurm with the real GPU in gres.conf; run --gres=gpu:1 and check confinement with nvidia-smi inside the job.B›
8. 31Write how a container job runs on Slurm with Pyxis and enroot, and when you would pick Slurm over Kubernetes for training.P›

The GPU Operator on a real GPU, scheduling rules on the laptop.

1. 32Install k3s on the GPU node and Helm-install the NVIDIA GPU Operator.B›
2. 33Confirm the device plugin advertises nvidia.com/gpu and read the labels GPU Feature Discovery added to the node.B›
3. 34Run a CUDA pod that requests one GPU, then a second that cannot be scheduled, and read its Pending reason.B›
4. 35Enable time-slicing and run two pods on one GPU. Then write when MIG is the better choice.B›
5. 36Switch the Operator to the MIG strategy and expose MIG slices as schedulable resources.B›
6. 37Deploy the DCGM exporter through the Operator and curl its metrics endpoint.B›
7. 38On the laptop k3d cluster, taint a node and schedule a pod with a toleration. Write how GPU nodes are kept for GPU jobs.A›
8. 39Install Kueue on k3d, create a ClusterQueue with a quota, and watch a job wait for quota and then run.A›
9. 40Write how the Network Operator gives pods RDMA access (device plugin, Multus, SR-IOV) and the pod spec lines it needs.P›
10. 41Write the difference between the driver container and a host-installed driver, and when each is used.P›

Prometheus, Grafana and Loki on the laptop, then pointed at the GPU node.

1. 42Helm-install kube-prometheus-stack on k3d and open Prometheus and Grafana.A›
2. 43Write three alert rules (node down, disk over 85 percent, pod crash-looping) and route them through Alertmanager.A›
3. 44Install Loki and Alloy, and run a LogQL query that finds a kubelet error.A›
4. 45Import Grafana dashboard 12239, the DCGM exporter dashboard, and study every panel.A›
5. 46Scrape the DCGM exporter on the GPU node from Prometheus and light up the dashboard under a training load.B›
6. 47Add an alert on Xid count and one on GPU temperature. Fire the temperature one by lowering its threshold.B›
7. 48Ship the GPU node kernel log to Loki and run the Xid-per-node LogQL query.B›
8. 49Design the monitoring for a 64-node GPU cluster: metrics, logs, dashboards, and which alerts page versus ticket.P›
9. 50Write an SLO and a burn-rate alert for job success and for throughput against the NCCL baseline.P›

RoCE in the free NVIDIA Air simulator; InfiniBand from saved command output, because no laptop has an HCA.

1. 51In NVIDIA Air, build a two-leaf, one-spine Cumulus Linux fabric and bring every link up.A›
2. 52Configure PFC and ECN for RoCE on the Air fabric and verify with the switch counters.A›
3. 53Shut a port, bring it back, and read the link-flap evidence in the switch log and counters.A›
4. 54Read saved ibstat, ibstatus and iblinkinfo output and find the one port trained at a lower speed or width.P›
5. 55Read a saved ibdiagnet report and list every port with rising symbol errors or link-downed counts.P›
6. 56Compare an ibnetdiscover topology file against a cabling plan and find the cable in the wrong leaf.P›
7. 57Read transceiver DOM data (temperature, Tx and Rx power) and decide which end of a link is faulty.P›
8. 58Write the counter sheet: SymbolError, LinkErrorRecovery, LinkDowned, PortRcvErrors, PortXmitDiscards, pause and ECN counters, with meaning and action.P›
9. 59Write the link-flap runbook: confirm from both ends, protect workloads, physical first, swap one part at a time, close out.P›

The number every later complaint is compared against.

1. 60Build nccl-tests on the GPU node and run all_reduce_perf across the GPUs or MIG slices.B›
2. 61Record the bus bandwidth as a baseline and compare it with the NVLink or PCIe figure for the system.B›
3. 62Run with NCCL_DEBUG=INFO and identify the transport and the NVLink or NIC path NCCL chose.B›
4. 63On two instances, run all_reduce_perf across nodes with NCCL_SOCKET_IFNAME set, and note the drop.C›
5. 64Write the bisect method for a cluster-wide all-reduce that is 40 percent below baseline.P›
6. 65Write a burn-in plan: DCGM diag level 3, HPL or a real training job, duration, and the pass criteria.P›

Everything through Git. The laptop is enough for all of it.

1. 66Write a Terraform module that creates a k3d cluster or a set of Docker containers; plan, apply, change a variable, destroy.A›
2. 67Create a VPC and one instance on AWS free tier with Terraform, with remote state and locking.A›
3. 68Output an Ansible inventory from Terraform and run a playbook against it.A›
4. 69Write an Ansible role for GPU host prep (section B) and prove it is idempotent with --check and a second run.A›
5. 70Put the Helm chart and playbooks in GitHub and add a GitHub Actions pipeline that lints and packages on every push.A›
6. 71Install Argo CD on k3d and deploy the chart from Git. Change a value in Git and watch it sync.A›
7. 72Roll back a bad release through Git and then through Argo CD. Write which you would use at 2 am.A›
8. 73Write a wave rollout playbook for a driver upgrade across 64 nodes: canary, one rack, the rest, with health gates.P›
9. 74Write how you stop a bad change reaching every node: review, CI, canary, gates, rollback.P›

No lab exists without licensed hardware, so this section is read and write. Say so in interviews.

1. 75Read the BCM administrator manual chapters on head nodes, software images and node categories, and draw the provisioning flow.P›
2. 76Write the cmsh commands you would use to list nodes, change a category and reboot a node into a new image.P›
3. 77Write the driver rollout answer for 256 nodes with BCM: clone the image, canary category, waves, rollback.P›
4. 78Compare BCM with MAAS, Warewulf and Metal3 in a one-page table.P›
5. 79Design the images and categories for a 64-node cluster with two GPU types and a login node tier.P›

Where the data lives and how a model is served, the two questions every company asks.

1. 80Run fio on the GPU node local NVMe and on an NFS mount, and record read and write throughput.B›
2. 81Write the checkpoint pattern for a 70B-parameter model: size, frequency, where it lands, and the storage throughput it needs.P›
3. 82Write what Lustre, GPFS, WEKA and VAST each give a GPU cluster, and when NFS over RDMA is enough.P›
4. 83Serve a small model with vLLM on the GPU node and measure tokens per second at two batch sizes.B›
5. 84Serve the same model with Triton or a NIM container and compare the metrics each exposes.B›
6. 85Write an autoscaling design for inference on Kubernetes using GPU utilisation and queue depth.P›

The documents the job description names. Write them once, then reuse them in every interview.

1. 86Write the GPU node replacement runbook: trigger, impact check, safe first actions, diagnosis, escalation, close-out.P›
2. 87Write the InfiniBand link failure runbook in the same shape.P›
3. 88Write a blameless post-incident review for a real or imagined outage: timeline, impact, causes, actions with owners.P›
4. 89Write a one-page executive update with RAG status, top risks, open actions and a path to green.P›
5. 90Prepare a ten-minute NVIDIA roadmap talk: current generation, the next two, networking, software.P›
6. 91Write four STAR stories mapped to the job description: technical lead, Kubernetes platform, runbooks, automation.P›
7. 92Write an onboarding guide for a new customer cluster: access, scheduler, storage, monitoring, who to call.P›

Ten minutes each, out loud, the way an interviewer will ask them.

1. 93Timed, 10 minutes: a multi-node training job runs at half the expected speed. Diagnose aloud.P›
2. 94Timed, 10 minutes: a node is NotReady and the kernel log shows Xid 79. Act and explain.P›
3. 95Timed, 10 minutes: one InfiniBand link flaps every few minutes. Walk through it.P›
4. 96Timed, 10 minutes: a pod that requests a GPU stays Pending.P›
5. 97Timed, 10 minutes: InfiniBand or Spectrum-X for a new 128-GPU cluster. Recommend and defend.P›
6. 98Timed, 15 minutes: design the monitoring for a 64-node cluster on a whiteboard.P›
7. 99Timed, 10 minutes: NCCL inside pods is using TCP sockets instead of InfiniBand. Why, and the fix.P›
8. 100Timed, 10 minutes: roll a new GPU driver to 256 nodes without a cluster-wide outage.P›

How to use itFour weeks, if that is what you have

Commands and product names were checked against the vendor documentation in October 2026. NVIDIA, Kubernetes and the observability projects release often, so confirm a flag against the current documentation before you rely on it in front of a customer.