A candidate wrote to me after his second attempt with a single line of frustration: he had found the stuck worker node, deleted it, and the upgrade still refused to move. That reflex is the most expensive habit on this objective, and objective 5.4 is very largely a test of whether you have unlearned it.

Three hours in Deleting is not a crash. Nothing has errored, no controller has given up, and every log you open will look calm. Cluster API is waiting, politely and indefinitely, for a drain that will never finish, and the object holding it up is not on that node at all.

Stall, failure and the difference that decides your next command

Broadcom documentation still calls this workflow a failed upgrade, and older material talks about restarting a failed update job. Both phrases are inherited from vSphere 7 and early vSphere 8, when a TanzuKubernetesCluster upgrade really was executed by a Kubernetes Job in the vmware-system-tkg namespace, and the remedy really was to bump that Job spec.backoffLimit so it retried. On VCF 9.0 with a ClusterClass based cluster there is no update job to restart. Reconciliation never stopped. It is looping, it is emitting the same condition message every few seconds, and it will keep doing so for days.

That distinction changes what recovery even means. Under a Job model, recovery is retry. Under a controller model, recovery is unblocking, and retry happens for free the moment the block is gone. Candidates who learned VKS on vSphere 7 walk into this objective carrying the Job mental model and reach for restart verbs that no longer exist, which is precisely the gap the objective wording quietly probes.

Worth saying once more in this Part, because it shapes every path you will type: VKS is the product formerly called TKG Service or TKGS. Broadcom renamed the product, not the plumbing. Recovery documentation for this objective lives under a URL path containing troubleshooting-tkg-service-clusters, the legacy update Jobs live in vmware-system-tkg, and the labels on them carry the run.tanzu.vmware.com prefix. Expect tkg in resource names and stop treating it as a sign you are reading stale docs.

There is a second split worth holding, and it is a version split. VKS 3.5 and 3.6 introduced system pre-checks that refuse to start an upgrade when known blockers are present, surfacing a SystemChecksSucceeded condition on the cluster with a message naming the offending PodDisruptionBudget or admission webhook. Our reference estate runs VKS 3.3.1, which predates those checks entirely. On 3.3.x the upgrade starts happily and then wedges silently at the first node it cannot drain, which is why this Part is a diagnosis exercise rather than a reading exercise. If your production estate is on 3.5 or later, the same faults present as a refusal to start rather than a stall, and you read the condition message instead of hunting for it.

flowchart TD
  A[Roll not progressing] --> B[Describe cluster and read conditions]
  B --> C{Which object is blocked}
  C -->|KCP waiting on preflight| D[Control plane node unhealthy or etcd short a member]
  C -->|Machine stuck Deleting| E[Drain is blocked]
  C -->|MachineDeployment on old version| F[Version drift between TKC and Cluster]
  E --> G{What refuses to evict}
  G -->|Pinniped cert agent| H[Create MachineDrainRule in Supervisor namespace]
  G -->|App pod with PDB| I[Fix PDB inside workload cluster]
  G -->|Volume still attached| J[Clear stale volumeattachment]
  D --> K[Restore control plane quorum then let KCP resume]
  F --> L[Align TKC version so Cluster follows]
  H --> M[Reconciliation resumes on its own]
  I --> M
  J --> M
  K --> M
  L --> M

Preflight that names the stall before you touch a node

Two contexts are in play and you will switch between them repeatedly, so fix them in your head now. Supervisor context is where the Cluster, Machine, MachineDeployment and MachineDrainRule objects live, inside the vSphere Namespace that owns the workload cluster. Workload cluster context is where pods, PodDisruptionBudgets and volumeattachments live. Cluster API only ever acts from the Supervisor side; it reaches into the workload cluster to drain, and when that reach fails it reports the failure as a Supervisor side condition. Reading a Supervisor condition and then reflexively fixing it in the workload cluster, or the reverse, is the single most common time sink here.

I use the vSphere plugin for kubectl for Supervisor login rather than a static kubeconfig, for one reason worth stating once: the plugin mints a short lived token bound to your identity provider session, so a recovery you perform is attributable in vCenter events afterwards. When a roll has been stuck across a shift handover, that audit trail is how you reconstruct who paused what.

Notice what that last line does not tell you. VERSION echoes the version you asked for, not the version your nodes run, exactly as established in Part 21, and PHASE reads Provisioned throughout a stall because the cluster genuinely is provisioned. Anyone triaging from those two columns concludes the upgrade finished. Conditions are the only honest surface.

Now the stall has a name. Drain is blocked on the Pinniped concierge certificate agent, a pod that exists only because this cluster has an external identity provider wired through it, per Part 8. That agent is a static helper that gets recreated on whichever node it is evicted from, so it restarts on the draining node and drain never converges. Broadcom documents this behaviour for VKS 3.3 and later, where a node that does not drain inside the configured drain timeout simply halts the upgrade rather than proceeding.

Numbered recovery procedure with verification at each step

Prerequisites before step 1: you hold edit permission on the vSphere Namespace that owns the cluster, you are logged in to both contexts, and you have confirmed from Part 25 that the Supervisor itself is not mid-update, because a Supervisor update in flight will hold workload cluster reconciliation and no amount of workload cluster surgery will help. Confirm that first with a single check of the Supervisor update status in vCenter under Supervisor Management, then Namespaces, then Updates.

Step 1, classify before you act. Run the describe from the preflight section and write down which condition is False and what its message names. Do not proceed on a guess. Roughly a third of the stalls I have worked turned out to be a different class from the one the on-call engineer assumed, and the assumed fix had already made the real one harder to see.

Step 2, repair a drain block from the Supervisor. For the Pinniped case above, the remedy is a MachineDrainRule that tells Cluster API it may skip that pod class. This object is created in the Supervisor, in the vSphere Namespace that owns the cluster, and this is the step people get wrong: the pod being skipped lives in the workload cluster, so the instinct is to apply the rule there, where the CRD does not even exist.

Two Machines briefly coexisting is correct and is what you want to see. A replacement entering Provisioning is proof that the MachineDeployment surge slot freed up, which only happens once the outgoing Machine actually finishes draining. Scope that rule tightly, because behavior Skip applied broadly will happily evict nothing and let a real workload be killed without notice.

Step 3, repair a PodDisruptionBudget block inside the workload cluster. A different signature, same symptom. When an application PDB permits zero disruptions, eviction is refused outright and the node parks in Ready,SchedulingDisabled instead of finishing deletion. Fixing this from the Supervisor is impossible, because the object is owned by the application team in the workload cluster.

Scaling up is the reversible move and it is what I reach for during a change window. Editing the PDB is the other option, and it is the one I avoid under pressure, because you are quietly rewriting an availability contract the application team wrote deliberately, and nobody remembers to put it back. Broadcom is explicit that a PDB with zero allowed disruptions should be resolved with the application owner rather than around them.

Step 4, clear a volume detach hang. Draining is only half of node deletion; volumes must detach too. If a volumeattachment lingers, usually after a node was powered off rather than drained, the Machine sits in Deleting with a clean drain and no obvious complaint. Check volumeattachments in the workload cluster and reconcile them against nodes that no longer exist, and be aware that a stale attachment referencing a deleted virtual machine image will need Broadcom support rather than a local fix.

Step 5, resolve version drift, and know when to stop. A nastier class appears when a second upgrade is started while the first is still rolling. Control plane Machines land on the new version while the MachineDeployment and its VSphereMachineTemplate stay on the old one, so replacement workers boot the old image and their antrea, kube-proxy and vsphere-csi-node pods enter ImagePullBackOff hunting images the new version expects. For a cluster still fronted by a TanzuKubernetesCluster object, confirm the TKC and the Cluster name the same Kubernetes Release and correct the TKC, because the TKC drives the Cluster and not the reverse. If the templates themselves are already skewed, that is a support case; Broadcom asks you to open one rather than hand-build a VSphereMachineTemplate, and building one by hand is how a stalled cluster becomes an unrecoverable one.

Back out, and a stall signature to remediation lookup

Back out on this objective is uncomfortable and you should say so plainly in any design review. There is no downgrade path for a workload cluster, as Part 21 established, so once control plane Machines have rolled onto a new Kubernetes Release your only true reversal is a restore. That is why the Velero work in Part 28 is a prerequisite for upgrade work and not an optional extra: a namespace level backup taken immediately before the roll is the difference between a bad afternoon and a rebuilt cluster. What you can do safely mid-stall is pause topology reconciliation on the Cluster object to stop further node churn while you investigate, then unpause once the block is cleared. Pausing buys thinking time; it does not undo anything already rolled.

This lookup is the artifact worth bookmarking from this Part. Match on the signature you can see without guessing, then act in the context named in column three.

Exam focus, objective 5.4

Objective checkpoint

Recovery order I follow before opening a case

Here is the war story, and it cost me a Saturday. A payments cluster stalled at 21:04 on a Friday during a routine Kubernetes Release bump. On-call read the Machine as hung, deleted it, and when the replacement also parked in Deleting, deleted that one too. By the time I picked it up at 09:00 the MachineDeployment had burned through four Machines, the drain evidence from the original had aged out of the event log, and the cluster was two workers short of its declared replica count with the surge slot permanently occupied. Actual repair, once I stopped and read the conditions, was one MachineDrainRule and 24 minutes. Total incident length was 13 hours, and 12 of them were self-inflicted by node deletion.

Verdict, and it contradicts most of what you will read in general Kubernetes forums: on VKS the stuck node is evidence, not the problem, and force deleting it destroys the only record of why the drain failed. Popular study advice on this objective leans on the legacy update Job and the backoffLimit patch, which is genuinely correct on vSphere 7 and on non ClusterClass clusters and genuinely useless on a VCF 9.0 estate. Learn both, and learn which is which, because an item that shows you a ClusterClass name in the output is telling you which world you are in.

Clean result looks like this, and it is worth checking all five: every Machine in the namespace reports Running on the target Kubernetes Release, MachineDeployment UPDATED equals REPLICAS as established in Part 21, TopologyReconciled and MachinesReady are both True, no node in the workload cluster sits in SchedulingDisabled, and any MachineDrainRule you added is either still needed or removed on purpose rather than forgotten.

Do this tonight in your own lab: take a Velero backup, apply a PodDisruptionBudget with minAvailable equal to your full replica count to a StatefulSet, then start a Kubernetes Release bump and time how long it takes you to reach the eviction event from a cold start. Then recover it twice, once by scaling and once by editing the budget, and note which one you would want to explain in a change review. Every component under this objective is covered in ordinary operational depth in the vSphere Kubernetes Service complete guide, and the identity provider wiring that creates the Pinniped drain block in the first place is worth revisiting there. Part 34 closes the series on optimising cluster performance with monitoring and scaling features, which is where the drain timeouts you just met get tuned deliberately rather than discovered at 21:04 on a Friday.

References

• Restart a Failed VKS Cluster Upgrade, Broadcom TechDocs
• Workload Cluster Upgrade Stuck on VKS 3.3 and higher due to Pinniped-Concierge Pods, Broadcom Knowledge Base 410900
• Workload Cluster Upgrade Stuck due to Node Stuck Deleting caused by PodDisruptionBudget, Broadcom Knowledge Base 345904
• Upgrade Stuck with Control Planes Upgraded but Worker Nodes Stuck due to MachineDeployment Version, Broadcom Knowledge Base 376919
• Overriding Software Misconfiguration Checks Blocking VKS Cluster Upgrade, Broadcom TechDocs

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts