, ,

VKS VM Class, Cluster Plan and Content Library Fault Diagnosis (VCAP-VKS Exam Series, Part 31)

Objective 5.2 packs five different fault families into a single line of blueprint text. Here is the five gate sweep I use to tell a VM class binding problem from a cluster plan certificate defect, with the real error strings and the surface that actually fixes each one.

VCAP-VKS Exam Series · Part 31 of 34
Who this is for: a vSphere admin or platform engineer who has worked through Part 30 and can recite the reading order without looking it up. Objective 5.2 in the published exam guide reads, verbatim, Troubleshoot VM class, cluster plans, content library, storage, and networking errors. Part 30 gave you the four gate chain across the whole stack. This Part stops inside gate one and takes its five inputs apart, one noun at a time, on the same VCF 9.0 estate.
Key takeaways: five nouns, five different owning surfaces, and only one of the five is repaired with kubectl alone. Cluster status reason strings name the object that failed to render, never the object that caused the failure, which is why VirtualMachineClassNotFound sends people off to build a VM class that already exists. Cluster plan faults on newer Supervisor Service builds are frequently a certificate rotation defect that returns on roughly a 60 day cycle and clears with a deployment restart, not a YAML edit. Content library faults are silent by design, because an unsynced or non compliant item does not raise an error, it simply fails to appear in kubectl get kr. Headline moves are kubectl get virtualmachineclasses -n NS, kubectl get cc -n svc-tkg-domain-cNN -o jsonpath and kubectl edit ippool -n NS.

Here is a line of output that has cost me more hours than any other single string on this objective.

"reason": "VirtualMachineClassNotFound @ /app-prod-9jd7q-gmmtk", "severity": "Error", "type": "ControlPlaneReady"

Read it plainly and it says a VM class is missing. Act on it plainly and you open vCenter and start building one. Then you find that guaranteed-medium already exists on this same vCenter, sized at 2 vCPU and 8Gi, and has existed since the estate was built eighteen months ago. Nothing in that string tells you the real fault is a binding checkbox two clicks deep in Workload Management, on a namespace whose name does not even appear in the error text.

Objective 5.2 is that same problem repeated five times with five different nouns. VM class, cluster plans, content library, storage, networking. Each one raises its error inside Kubernetes and then gets repaired somewhere that is not Kubernetes, or gets repaired by restarting a component the error never mentions. Memorising the strings is not sufficient, and I will argue at the end of this Part that memorising them is actively counterproductive. What carries you through is a mapping from noun to owning surface, plus a fixed order for walking it.

One naming note, said once here and not repeated. VKS is the product formerly called TKG Service or TKGS, and that rename stops at the marketing layer. You will read Cluster API logs in a namespace called svc-tkg-domain-cNN, list Kubernetes releases with kubectl get kr where the short name descends from the tkg era, and find the stock cluster plan carrying a ClusterClass named tanzukubernetescluster. None of that is legacy output. It is current VCF 9.0 output, and both the product and the exam print it exactly as the CLI does.

Minutes to root cause, five fault classes Median of three induced runs each on the series lab estate, VCF 9.0 with three vSphere Zones Unstructured attempt Gate sweep 0 20 40 60 41 3 VM class 70 6 Cluster plan 26 4 Content library 33 8 Storage zone 52 11 Stale subnet Cluster plan faults punish guessing hardest, because the error names a manifest and the fix is a pod restart.

Preflight before you touch a cluster

Every timing in that chart came from inducing a fault deliberately, walking away, and coming back cold. What separates the two bars is not knowledge, it is whether I established four things before forming an opinion. Which namespace the cluster lives in, which svc-tkg domain namespace serves it, which Supervisor Service version is installed, and which zone each node landed in. Skip those and every subsequent command is a guess dressed up as a diagnostic.

# Estate every command below was run against # VCF 9.0 workload domain, three vSphere Zones, NSX VPC networking, Avi in the load balancer path # VKS Supervisor Service 3.3.2 in the series lab # cluster plan fault reproduced separately on a 3.4.1 Supervisor Service estate # kubectl v1.32.5, kubectl-vsphere plugin 9.0, VCF CLI 9.0 # password is read from KUBECTL_VSPHERE_PASSWORD in the shell environment, never typed inline kubectl vsphere login –server=$SUPERVISOR_VIP –vsphere-username $SV_USER –insecure-skip-tls-verify export NS=vvks-lab-prod export SVCNS=$(kubectl get ns -o name | grep svc-tkg-domain | cut -d/ -f2) echo $SVCNS svc-tkg-domain-c1009 kubectl get supervisorservice -A | grep -i vks vmware-system-supervisor-services vks 3.3.2 Ready

Derive that svc-tkg domain namespace rather than recalling it. On a multi cluster workload domain the suffix tracks the vCenter cluster moref, so an estate with three vSphere Zones can carry more than one of them, and typing the wrong suffix produces perfectly formatted output about a cluster you are not troubleshooting.

Numbered fault isolation procedure

Walk these five gates in order. Order matters because gates one and three are cheap and unconditional, while gates four and five require you to know which zone a workload landed in, and that knowledge only exists once machines have rendered. A fault at gate one makes gates four and five unanswerable, so a sweep that starts at networking wastes its first twenty minutes proving that a cluster with no nodes has no node networking.

Gate 1. VM class binding on the namespace

A cluster requests a VM class by name in its topology variables. VM Operator resolves that name against VirtualMachineClass objects present in the namespace, and a class that exists globally but is not bound to the namespace does not exist as far as the render is concerned.

kubectl get cluster app-prod -n $NS -o jsonpath='{.status.conditions}’ | jq -r ‘.[] | select(.severity=="Error") | .reason’ VirtualMachineClassNotFound @ /app-prod-9jd7q-gmmtk kubectl get cluster app-prod -n $NS -o jsonpath='{.spec.topology.variables}’ | jq -r ‘.[] | select(.name=="vmClass") | .value’ guaranteed-medium kubectl get virtualmachineclasses -n $NS NAME CPU MEMORY best-effort-large 4 16Gi best-effort-medium 2 8Gi best-effort-small 2 4Gi guaranteed-2xlarge 8 64Gi guaranteed-xsmall 2 2Gi # guaranteed-medium is requested, is absent here, and exists globally in vCenter # fix in vCenter: Workload Management, the namespace, VM Service, Manage VM Classes kubectl get virtualmachineclasses -n $NS | grep guaranteed-medium guaranteed-medium 2 8Gi

Two details are worth carrying into the exam. First, the VM Operator controller log carries the same fault in a form that names the namespace explicitly, which the cluster condition does not. Second, this failure is also the reason a cluster can sit at Provisioned rather than Provisioning, because the Cluster object rendered fine and only the machine underneath it failed, per Broadcom knowledge article 402985.

Gate 2. Cluster plan reconciliation

Cluster plans surface as ClusterClass objects in the svc-tkg domain namespace, and every cluster operation waits on the plan reconciling its variables. When variable discovery fails, scaling, creation and upgrade all stop, and nothing in the cluster you are staring at is wrong.

kubectl describe cluster app-prod -n $NS | grep -A3 TopologyReconciled Message: ClusterClass is not successfully reconciled: status of VariablesReconciled condition on ClusterClass must be "True" Reason: ReconcileFailed Status: False Type: TopologyReconciled kubectl get cc -n $SVCNS builtin-generic-v3.3.0 -o jsonpath='{.status.conditions}’ | jq -r ‘.[] | select(.type=="VariablesReconciled") | .message’ VariableDiscovery failed: failed to call DiscoverVariables for patch default: failed to call extension handler "discover-variables.runtime-extension": http call failed: Post "https://runtime-extension-webhook-service.$SVCNS.svc:443/hooks.runtime.cluster.x-k8s.io/v1alpha1/discovervariables/discover-variables?timeout=10s": tls: failed to verify certificate: x509: certificate signed by unknown authority kubectl rollout restart deploy runtime-extension-controller-manager -n $SVCNS kubectl rollout restart deploy capi-controller-manager -n $SVCNS deployment.apps/runtime-extension-controller-manager restarted deployment.apps/capi-controller-manager restarted kubectl describe cluster app-prod -n $NS | grep -A1 TopologyReconciled Reason: Available Status: True
Production gotcha worth a calendar entry: on Supervisor Service builds 3.4.1 and later, client certificate rotation for the runtime extension controller is not handled cleanly, so the secret and the running pod drift apart and variable discovery fails with an unknown authority error. Broadcom knowledge article 424003 states this recurs on roughly a 60 day cadence until you reach VKS 3.5.1 or 3.6.0, and the repair is a deployment restart taking under two minutes. My recommendation for any estate sitting on 3.4.x is a standing 55 day reminder to check TopologyReconciled across every workload cluster, because the fault is invisible until somebody tries to scale and the first symptom a platform team usually reports is that autoscaling stopped working.

Gate 3. Content library and Kubernetes release availability

Kubernetes releases reach a Supervisor through a content library, subscribed or local. This gate is the quietest of the five because a release that failed to synchronise or that carries a non compliant template raises no error at all. It is simply absent, and absence reads exactly like a version that was never published.

kubectl get kr -A | grep -E ‘NAME|1.32.7’ NAME VERSION READY COMPATIBLE CREATED UPDATES AVAILABLE # header only, so the release you targeted is not present at all kubectl get kr -A –no-headers | wc -l 9 kubectl get kr -A –no-headers | awk ‘{print $1, $3, $4}’ v1.31.4—vmware.1-fips True True v1.32.0—vmware.2-fips True True v1.32.5—vmware.1-fips True True # next check is in the vSphere Client, not in kubectl: # Content Libraries, the VKS library, the item for 1.32.7, is it marked Security Compliant

Broadcom knowledge article 425905 covers the case where a library item is present but corrupt or not security compliant, and the Supervisor therefore never registers it as a Kubernetes release. Note also the COMPATIBLE column. A release that lists as READY True and COMPATIBLE False is available but not a legal upgrade target for the plan you are running, which is a different fault with an identical looking symptom for anybody who reads only the first column.

Gate 4. Storage policy reach across zones

Storage classes inside a namespace are generated from storage policies assigned to that namespace, so a missing class is a vCenter assignment problem. On a zonal Supervisor there is a nastier variant. Assignment succeeds, the class appears, and volumes still fail, because the policy is only compatible with datastores in two of the three clusters that make up the zonal topology.

# on the workload cluster context kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE vks-default csi.vsphere.vmware.com Delete WaitForFirstConsumer vks-high-performance csi.vsphere.vmware.com Delete WaitForFirstConsumer kubectl get pvc -n app NAME STATUS VOLUME CAPACITY STORAGECLASS pgdata-postgres-0 Bound pvc-8c 20Gi vks-high-performance pgdata-postgres-1 Bound pvc-1a 20Gi vks-high-performance pgdata-postgres-2 Pending vks-high-performance # the class exists and two identical claims bound, so this is not a missing class # find the zone the third replica was scheduled into kubectl get pod postgres-2 -n app -o jsonpath='{.spec.nodeName}’ app-prod-md0-x4k2z kubectl get node app-prod-md0-x4k2z -o jsonpath='{.metadata.labels.topology.kubernetes.io/zone}’ az3 # next check is in vCenter: VM Storage Policies, the high performance policy, # Check Compliance, and confirm compatible datastores exist in the az3 cluster

Read the pattern rather than the message here. The provisioning event text from vSphere CSI is generic and rarely names a zone, so what tells you the answer is that two claims from the same StatefulSet bound and the third did not. A policy built against a vSAN storage tag that only two of three clusters carry produces exactly this, and it will pass every test you run in az1 and az2.

Gate 5. Node networking and stale subnet entries

Last gate, and the one most likely to be blamed first. A machine that reaches Provisioned and stops, with a healthy NCP and no NSX alarms, is usually a leftover from a rolling redeployment rather than a networking outage.

# on the Supervisor context kubectl get machine -n $NS NAME CLUSTER NODENAME PHASE AGE VERSION app-prod-md0-x4k2z app-prod Provisioned 38m v1.32.5+vmware.1-fips kubectl get pods -n vmware-system-nsx NAME READY STATUS RESTARTS AGE nsx-ncp-6c9f47b8d5-q7twm 1/1 Running 0 21d # on the workload cluster context kubectl describe node app-prod-md0-x4k2z | grep Taints Taints: node.kubernetes.io/network-unavailable:NoSchedule # back on the Supervisor context kubectl describe ippool -n $NS app-prod-ippool | sed -n ‘/subnets:/,$p’ subnets: – ipFamily: ipv4 name: app-prod-md0-77b9q prefixLength: 24 – ipFamily: ipv4 name: app-prod-md0-x4k2z prefixLength: 24 # app-prod-md0-77b9q is not in kubectl get machine output, so that entry is stale kubectl edit ippool -n $NS app-prod-ippool

Remove only the entry naming a machine that no longer exists, save, and the cleanup propagates to the NSX side. Broadcom knowledge article 402228 documents this as a vSphere CPI defect where deletion requests for the subnet are not sent when a node is replaced. Deleting a live entry by mistake is destructive, so compare against kubectl get machine output every time rather than trusting the naming pattern.

Verification, rollback and a clean end state

Each gate has its own proof, and none of them is the cluster reporting Ready, because a cluster can report Ready while carrying a fault that only bites on the next scale operation. Gate one is green when kubectl get virtualmachineclasses in the namespace lists the exact class named in the topology variables. Gate two is green when TopologyReconciled shows Status True with Reason Available on every workload cluster served by that svc-tkg domain namespace, not just the one you were asked about. Gate three is green when kubectl get kr lists your target version with READY True and COMPATIBLE True. Gate four is green when a test claim binds in every zone, which means three claims, not one. Gate five is green when the IPPool subnet list and kubectl get machine agree on names, one for one.

Rollback is straightforward for four of the five and genuinely risky for one. Binding a VM class, assigning a storage policy and restarting a controller deployment are all reversible with no data consequence, so back out by reversing the same action on the same surface. Editing an IPPool is the exception. There is no undo, and removing a subnet entry that belongs to a live node strands that node. Before saving that edit, capture the object with kubectl get ippool -n NS NAME -o yaml written to a file, so a mistaken removal can be reapplied. I have needed that file exactly once, which was enough to make it a habit.

Fault to owner lookup

This table is the artifact worth keeping. Column three is the point of the whole exercise, because in four of these five rows the object that owns the fault is not the object the error names, and in three of them the repair surface is not kubectl at all.

NounWhat you seeObject that owns itSurface that fixes it
VM classVirtualMachineClassNotFound in cluster status, machine stuck, no VirtualMachine rendersVirtualMachineClass binding on the vSphere Namespace, not the global classvCenter, Workload Management, namespace, VM Service, Manage VM Classes. Article 402985
Cluster plansTopologyReconciled False, VariablesReconciled must be True, all cluster operations refuse to startRuntime extension webhook certificate in svc-tkg-domain-cNN, drifted after rotationkubectl rollout restart runtime-extension-controller-manager, then capi-controller-manager. Article 424003
Content libraryTarget version simply absent from kubectl get kr, no error raised anywhereLibrary item sync state or security compliance flag on the templatevSphere Client, Content Libraries, confirm the item is Security Compliant. Article 425905
StorageOne PVC Pending while identical claims from the same StatefulSet bind normallyStorage policy compatibility in one cluster of the zonal topologyvCenter, VM Storage Policies, check compliance per cluster, then reassign to the namespace
NetworkingMachine stuck Provisioned, node taint network-unavailable, NCP healthy, no NSX alarmsStale subnet entry in the IPPool left behind by a rolling redeploymentkubectl edit ippool on the Supervisor, remove only entries naming missing machines. Article 402228
flowchart TD
  A[Cluster operation not completing] --> B{VM class bound to namespace}
  B -->|No| C[Bind class in VM Service, fix is in vCenter]
  B -->|Yes| D{TopologyReconciled True}
  D -->|No| E[Restart runtime extension and capi controllers]
  D -->|Yes| F{Target release listed by kubectl get kr}
  F -->|No| G[Check content library item compliance]
  F -->|Yes| H{Claims bind in every zone}
  H -->|No| I[Check policy compatibility per cluster]
  H -->|Yes| J{IPPool matches machine list}
  J -->|No| K[Remove stale subnet entry only]
  J -->|Yes| L[Fault is outside objective 5.2, return to Part 30 chain]
Five gates in cost order. Every No branch names a surface, and three of the five surfaces are not kubectl.
Scope note: general Kubernetes scheduling behaviour and NSX packet level analysis are owned elsewhere in the catalogue and get one clause each here. For the component mechanics behind these five nouns see the VKS Series, and for VPC, segment and IP block behaviour see the NSX Series. Anyone arriving from a Pivotal estate should start at the TKGI to VKS Series, because none of these five objects exist in the same shape there.

Exam focus, objective 5.2

Objective 5.2, what it actually tests: whether you can separate five fault families that all present as a cluster operation that will not finish, and whether you know which of them are repaired outside Kubernetes. Expect scenario style multiple choice with a status condition or an event line pasted into the stem, matching items pairing a symptom with a component or a vCenter screen, and build list items asking you to order a set of checks. Point and click and hot area items are a natural fit for the VM Service and VM Storage Policies screens, since two of these five repairs live there. Multiple selection items tend to ask which two conditions together identify a fault, which is a discrimination question rather than a recall question. The single trap that catches experienced admins: treating the reason string as the diagnosis. VirtualMachineClassNotFound reads like a missing object and is almost always a missing binding, and a ClusterClass error reads like a manifest problem and is often an expired certificate on a controller you were never looking at. If an answer option repairs the object named in the error, be suspicious of it.

Objective checkpoint

Three original questions, written from the published objective wording:

1. A cluster reports VirtualMachineClassNotFound, and the class named in the error is visible in vCenter under VM Classes with the correct CPU and memory sizing. What is the next action? Answer: bind the class to the vSphere Namespace through VM Service, Manage VM Classes. VM Operator resolves the class name against namespace scoped objects, so a globally defined class that is not bound to the namespace cannot be selected, and creating a second class only produces a duplicate.

2. Scaling a worker pool does nothing, MachineDeployment stays in Running phase with unchanged replicas, and the cluster reports that ClusterClass variables are not reconciled. Where does the fault live? Answer: with the runtime extension controller in the svc-tkg domain namespace, whose serving certificate no longer matches its secret. Variable discovery is a webhook call, a TLS failure on that call blocks every topology driven operation, and the cluster manifest is untouched.

3. A three replica StatefulSet on a zonal Supervisor has two persistent volume claims Bound and one Pending, and the storage class is present in the namespace. Which check discriminates fastest? Answer: identify the zone of the node hosting the pending pod, then verify storage policy compatibility in that cluster. A missing class would fail all three claims equally, so a split result points at policy reach across the zonal topology rather than at namespace assignment.

Sweep order I trust over intuition

My worst run at this objective was a customer estate where autoscaling had stopped and nobody could say when. Cluster looked healthy, nodes were up, workloads served traffic. Scaling simply did not happen. I spent seventy minutes on it and made three separate edits to the cluster manifest, because the condition message said ClusterClass and my instinct said manifest. I bumped a variable, reverted it, changed the machine deployment replica count by hand, reverted that too. Nothing moved. What eventually fixed it was a single rollout restart of a controller deployment I had not opened once, and the whole repair took ninety seconds. Then it came back sixty one days later, on a Sunday, and by then I recognised it in four minutes.

So here is my verdict, and it cuts against the most common piece of study advice for this section. Popular guidance says to build a flashcard deck of error strings for troubleshooting objectives. For objective 5.2 that deck will actively mislead you, because four of these five errors name the wrong object on purpose, in the sense that a controller can only report what it failed to resolve, never why resolution failed. Learn the ownership map instead. Noun to owning object to repair surface, five rows, the table above. That map survives version changes and a string deck does not, and it is also what an item writer is testing when they paste a condition message into a stem and offer you four plausible repairs.

Tonight, in your own lab, induce two of these deliberately. Unbind a VM class from a namespace with a running cluster, then try to scale it, and time yourself from symptom to the vCenter checkbox. Then create a storage policy that only matches datastore tags in two of your three clusters, assign it, and deploy a three replica StatefulSet against it. Twenty five minutes of breakage buys you the two hardest rows in the table, in a form you will not forget under exam pressure. Next Part stays in Section 5 and moves past provisioning into what happens after the cluster is fine and the container is not, with objective 5.3 on container deployment, registry and trusted CA errors.

VCAP-VKS Exam Series · Part 31 of 34
« Previous: Part 30  |  Guide  |  VKS Series  |  Next: Part 32 »

References

About The Author


Discover more from Journal of Intelligent Infrastructure

Subscribe to get the latest posts sent to your email.

Leave a Reply

Your email address will not be published. Required fields are marked *

Architect’s Toolkit

About the Author

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

Discover more from Journal of Intelligent Infrastructure

Subscribe now to keep reading and get access to the full archive.

Continue reading