A candidate on a coaching call asked me why his cluster had sat at Provisioning for forty minutes while every Supervisor health check he ran came back clean. He had spent all forty of those minutes tailing controller logs in vmware-system-capw. On his VCF 9.0 estate that namespace still existed and still had running pods in it, and not one of them had touched his cluster. Objective 5.1 is, more than anything, about not losing forty minutes that way.

Three layers, and where a provisioning failure actually lives

Every failure in this objective sits in one of three layers, and each layer produces symptoms that look like they belong to the layer below it. Layer one is the vSphere Namespace, which owns storage policy assignment, VM class binding, Kubernetes release availability through the content library, and zone membership. Layer two is the controller set, which reads that namespace and renders a Cluster into VSphereCluster, VirtualNetwork, KCP, MD and finally VirtualMachine objects. Layer three is the running cluster itself, where the VIP either answers on 6443 or does not.

Symptoms travel downward and causes sit upward. A namespace with no storage policy assigned does not announce itself as a namespace problem. It announces itself as a PersistentVolumeClaim that sits Pending forever with an event about waiting for a volume, four layers away from the vCenter checkbox that caused it. A namespace with no bound VM class does not say so either. It says nothing at all, because kubectl get vm,machine -n ns-payments returns an empty list and an empty list looks like the controller has not started yet rather than like the controller has nothing valid to render.

That asymmetry is why the reading order matters so much. If you begin at layer three you will spend twenty minutes proving that a cluster which never had node virtual machines is unreachable, which was never in doubt. If you begin at layer one, three commands tell you whether the platform could possibly have built what you asked for. This is the Supervisor premise of the whole series in its most literal form: a VKS cluster inherits everything, so a VKS cluster failure is usually a Supervisor object failure wearing a Kubernetes costume.

flowchart TD
  A[vSphere Namespace on Supervisor] --> B[Storage policy, VM class, VKr, zones]
  B --> C{Namespace complete}
  C -->|No| D[No machine objects render at all]
  C -->|Yes| E[Controllers in svc tkg domain cNN]
  E --> F[VSphereCluster and VirtualNetwork]
  F --> G[KCP and MachineDeployment]
  G --> H[VirtualMachine objects and node VMs]
  H --> I{VIP answers on 6443}
  I -->|No| J[Nodes never join, KCP shows Unavailable]
  I -->|Yes| K[Cluster reports Ready]
  J --> L[Check reachability, then restart controllers]

Preflight and a controller namespace that misleads everyone

Before any diagnosis, establish two things: which Supervisor context you are in, and which namespace holds the controllers on this particular estate. That second one is not cosmetic. Broadcom documentation and a decade of community posts point at vmware-system-capw, and on a VCF 9.0 Supervisor with VKS delivered as a Supervisor Service, vmware-system-capw is still present, still has pods, and is no longer the namespace that reconciles your workload clusters. Cluster API for VKS moved into a per domain namespace named svc-tkg-domain-cNN, where NN is the vCenter cluster domain identifier and varies per environment.

Write that domain identifier on a sticky note for the length of the session. Every controller log command in this Part needs it, and typing the wrong one returns a clean empty result rather than an error, which is the worst possible outcome when you are under time pressure. A second preflight worth doing once: confirm the Supervisor control plane itself is healthy before blaming a workload cluster. Three control plane virtual machines, a floating IP that answers, and a namespace list that returns promptly are enough. If the Supervisor is degraded, nothing downstream is diagnosable and you are in a different objective.

Numbered diagnostic procedure

1. Prove the namespace can build what you asked for

Three commands, always in this order, always before anything else. A VM class list, a storage class list, and a Kubernetes release list. If any of the three comes back empty or missing the item your manifest names, stop diagnosing and go fix the namespace in vCenter. Nothing downstream will improve.

Note what that failure did not say. It did not mention vCenter, a storage policy, or a namespace. A candidate who has memorised Kubernetes storage will start writing a StorageClass manifest, which on a Supervisor namespace is both futile and unsupported, because storage classes there are projections of assigned vSphere storage policies rather than objects you author.

2. Read the machine chain top to bottom

One command shows you exactly how far the render got. An empty result at any stage tells you which controller stopped and, by extension, which input it was missing.

That empty pair of quotes is the entire diagnosis. An infrastructureRef block was applied with an apiVersion, a kind and a namespace but no name key, so the controller dutifully searched for a VSphereCluster called nothing. Broadcom documents this exact string in knowledge article 416254. Remove or correct the malformed key and reapply. Nobody needs a log for this one, which is the point of reading object status before reading controller output.

3. Interrogate connectivity from both sides of the VIP

When machines exist and nodes never join, the question is whether the control plane endpoint answers, and from where. Check it from the Supervisor context first, because a mismatch between the recorded endpoints and the actual node addresses is an infrastructure fault, while agreement plus a timeout is almost always a stale controller.

Two checks separate the two causes. Confirm the cluster is not paused, because a paused cluster produces silence that looks identical to a broken one. Then reach the VIP from a workload cluster control plane virtual machine with a plain curl on 6443. If that succeeds and kubectl against the workload cluster is healthy, the platform is fine and the controller cache is not. Broadcom knowledge article 394554 documents this as a known Cluster API sync fault with a rollout restart as the resolution.

4. Read controller logs, in the correct namespace

Two habits save time here. Controllers run as deployments with leader election, so a replica that is not the leader logs one line about acquiring a lease and then goes quiet; targeting the deployment rather than a pod avoids reading a follower for ten minutes. And grep for your namespace name before you read anything, because a busy Supervisor produces reconcile chatter for every cluster on the estate.

5. Escalate to the control plane virtual machine only when object status is silent

A Supervisor that never reaches Ready, or a workload node that boots and then does nothing, is the one case where you leave the API. Retrieve the Supervisor floating IP and credentials from vCenter, then read the container runtime directly.

That eth1 line is worth committing to memory. A Supervisor control plane virtual machine needs two virtual network adapters, eth0 for management and eth1 for the primary workload network. When the agent manager attaches only one, the API server cannot bind to its workload address, etcd never answers, and the Namespaces tab in vCenter sits at Configuring indefinitely. Broadcom knowledge article 430866 is explicit that editing the virtual machine hardware by hand does not fix it and a clean redeployment is required, after confirming both IP pools have at least five free addresses per cluster: three for control plane nodes, one for the VIP, one for upgrade headroom.

Verification, rollback and what green looks like

Verification in this objective means proving each gate rather than trusting a single phase field. Green is four things: every VKr you reference reports READY True and COMPATIBLE True, storage class and VM class lists in the namespace contain the names your manifest uses, KCP reports READY equal to REPLICAS with API SERVER AVAILABLE true, and the Cluster phase reads Provisioned with a control plane endpoint that answers curl on 6443 from a workload node.

Rollback needs honesty. Most steps in this objective are read only and carry no rollback burden at all, which is a good reason to exhaust them before acting. Of the actions that do change state, a controller rollout restart is safe and self healing, since a restarted controller re-reads live state. Deleting a stuck PersistentVolumeClaim to clear drifted quota accounting is not reversible, so capture the claim to a file first and delete it from the layer that created it, workload cluster claims from inside the workload cluster and Supervisor claims from the Supervisor. Unassigning and reassigning a storage policy on a namespace to rebuild the quota custom resources will interrupt provisioning for anything using that policy, so it belongs in a window, not in a triage. And a Supervisor redeploy, which is the documented answer to the single vNIC fault, has no rollback whatsoever, which is why it is the last thing on the list and never the first guess.

Symptom to layer lookup

This table is the artifact worth returning to. Every row is an error string I have seen printed on a real estate, matched to the layer that owns it and the action that clears it. Print it, keep it beside your lab, and add rows as you break things.

Exam focus, objective 5.1

Objective checkpoint

Diagnostic order I would take into the exam

My own version of the forty minute story is worse, because I had no excuse. A customer cluster refused to scale past two workers on a Thursday afternoon. I opened controller logs in vmware-system-capw, found a leader election line, decided the controller was wedged, and restarted it. Nothing changed, so I restarted it again. Fifty five minutes in, a colleague asked which namespace I was reading. Once I moved to svc-tkg-domain-c1009 the answer appeared in the first twenty lines: the namespace had a VM class quota that permitted exactly two guaranteed-medium machines. Time to fix after that: four minutes, in vCenter, on a screen I had not opened once. Fifty five minutes lost to a namespace name I had typed from memory.

So here is my verdict on how to carry this objective. Popular study advice for troubleshooting objectives is to build a command list and drill it. I think that is the wrong investment, and my own worst hour is the evidence. I knew every command I needed that Thursday. What I lacked was an order and a habit of verifying my assumptions about where things live. Learn the four gate chain instead: namespace inputs, render objects, endpoint reachability, controller state. Learn to derive the domain namespace rather than recall it. Those two things transfer to versions that have not shipped yet, while a memorised namespace name has already expired once.

Tonight, in your own lab, do this. Pick a namespace, remove one storage policy assignment, and try to provision a cluster that references it. Time how long it takes you to get from the first symptom to the vCenter checkbox. Then put the policy back, unbind a VM class, and do it again. Two induced failures, twenty minutes, and you will own the top half of this objective in a way no command list delivers. Next Part stays in Section 5 and goes deeper into the namespace inputs themselves: VM classes, cluster plans, content library and storage and networking faults under objective 5.2.

References

• New workload cluster nodes not creating due to CAPI pods failing to reach the cluster VIP, Broadcom knowledge article 394554
• Supervisor deployment stuck in Provisioning or Configuring, Broadcom knowledge article 430866
• Newly created VKS cluster failing to complete provisioning, Broadcom knowledge article 416254
• PVC creation fails with insufficient storage quota for storage policy, Broadcom knowledge article 391096
• VMware Cloud Foundation VKS Administrator exam guide, 3V0-24.25

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts