Forty seconds after I saved a manifest edit on vks-lifecycle-01, kubectl reported the cluster VERSION as v1.32.0+vmware.6-fips. Not one node had restarted. No image had been pulled. That column is reading back the number I typed into spec.topology.version, not the Kubernetes version any node is actually running, and a candidate who verifies an update from it will be wrong in both directions, in the exam and in a change window.

Last part we drove vks-lifecycle-01 through provisioning, monitoring, scaling and deletion. This part we take that same cluster from Kubernetes 1.31.4 to 1.32.0, then change two configuration variables underneath it, and watch how differently the platform behaves in each case. Same three zone VCF 9.0 estate throughout, same namespace, same node shapes.

Preflight before editing a cluster manifest

Two facts decide whether an update will even begin. Whether a compatible newer VKr exists in the namespace, and whether the cluster carries a condition that blocks updates. Both are readable in about fifteen seconds, and skipping them is how people end up staring at a manifest edit that produced nothing at all.

Note the resource name in the next block. From VKS 3.2 onward the release object is kubernetesreleases. Below 3.2 it is tanzukubernetesreleases. Both exist in the wild, both appear in Broadcom documentation, and an item that shows you one of them is testing whether you know which platform generation you are looking at.

UpdatesAvailable is the condition worth internalising. It is computed against compatibility, not against what happens to be sitting in your content library, so it will hide a release the library carries but the cluster cannot legally move to. If UpdatesAvailable is absent or empty, editing the version field will be silently ignored by the topology controller and you will lose twenty minutes wondering why.

Rolling update mechanics on VKS 3.3

Two controllers share the work, an add ons controller and a cluster controller, and between them a roll has three stages that always run in this order: add ons, control plane, worker nodes. Each stage is gated by prechecks so a later stage cannot start until the earlier one has progressed far enough, and a stage is skipped entirely when nothing in it changed. A change that only touches workers therefore never disturbs your control plane.

Per node, replacement is additive first and destructive second. A new node is created and joined at the target configuration, then the old node is cordoned and drained, and only once every pod has left is that node deleted. If a pod cannot be evicted, the old node sits cordoned forever and the roll does not advance. Cluster API respects PodDisruptionBudgets absolutely, which is a feature until somebody writes one badly.

flowchart TD
  A[Spec edit or system change] --> B[Add ons controller reconciles packages]
  B --> C[Control plane nodes replaced first]
  C --> D[New node joins at target config]
  D --> E[Old node cordoned and drained]
  E --> F{Every pod evictable}
  F -- yes --> G[Old node deleted, next node begins]
  F -- no --> H[Node stays cordoned, roll stalls here]
  G --> I[Worker node pools roll in order]
  I --> J[Cluster phase returns to Provisioned]

One consequence gets people fired. Pods not governed by a replication controller, meaning bare pods that are not part of a Deployment, ReplicaSet or StatefulSet, are deleted during the worker drain and never recreated. Somebody kubectl run a debug pod with a mounted volume, walk away, and an overnight update will remove it without a trace in the application logs.

Version updates by editing the VKr

For a v1beta1 or v1beta2 cluster, a Kubernetes version change is a single field. Edit spec.topology.version to the name of the target VKr, save, and the topology controller does the rest. Two details trip candidates up. First, the value in the manifest uses the VKr object name with three hyphens before vmware, while every status field and printed column uses the plus sign form. Second, the TKR_DATA variable in the same manifest updates itself, and hand editing it is a good way to produce a cluster that will not reconcile.

On my estate that edit produced 57 minutes of work. Three control plane nodes replaced sequentially at roughly 11 minutes each, then three workers at roughly 8 minutes each, strictly one worker at a time beginning with the zone A node pool. Nothing overlapped. If you have two node pools and you assumed a version update would roll them in parallel to save time, it will not, and provisioning identical extra node pools to game that is a documented waste of capacity.

Configuration changes that roll nodes

Objective 4.6 pairs rolling updates with configuration changes for a reason: a configuration variable edit is a rolling update, it just has a different shape. VM class and storage class are the two the blueprint cares about, and both replace nodes rather than reconfiguring them in place. What changes is the concurrency. During a configuration variable update, control plane nodes go first as usual, and then one worker per node pool rolls simultaneously. Two node pools means two workers replaced at once, which is why the storage class change on the same cluster finished in 34 minutes where the version change took 57.

Here is the artifact worth keeping, a change to blast radius lookup. Print it, tape it inside the change ticket template, and stop guessing how long a window needs to be.

Verification signals worth trusting

Go back to the opener. Cluster VERSION reflects your spec, so it is a statement of intent. Three signals report reality: the UPDATED column against REPLICAS on KubeadmControlPlane and MachineDeployment, the TopologyReconciled condition, and Machine object ages. Machine age is the crude one I trust most, because a node that was not replaced has an age older than the moment you saved the edit, and no controller can lie about that.

Stall remediation and fallback paths

A stalled roll looks identical to a slow one for the first ten minutes, which is exactly how long people wait before they start clicking things they should not. Two classes of stall dominate. Prechecks that refuse to let the update begin, and drains that refuse to let a node leave.

ALLOWED DISRUPTIONS reading 0 is the whole story. A PDB with minAvailable set to 3 on a StatefulSet with exactly 3 replicas grants zero eviction budget forever, so Cluster API correctly refuses to evict, and correctly waits. Change minAvailable to 2, or express it as maxUnavailable, and the eviction happens within seconds. Part 20 met this same wall from the scaling side; the difference here is that a version update parks the whole cluster behind it rather than one node pool.

Rollback deserves a blunt sentence. There is no version downgrade. Editing spec.topology.version back to the older VKr is refused, and if you find a way to force it you will produce a cluster nobody supports. Your genuine fallback positions are, in order: stop the roll from progressing further by fixing whatever is blocking it and leaving the cluster in a mixed version state, which is supported for the duration of an update; restore workload state from Velero into a fresh cluster; or, for a configuration variable change only, edit the variable back to its previous value and accept a second full roll. Configuration is reversible. Kubernetes versions are not.

My worst version of this cost 4 hours and 12 minutes. I started a 1.31 to 1.32 update on a Thursday evening, watched the control plane roll cleanly, saw the first worker come up, and went to bed. At 06:20 the cluster still read Provisioned, VERSION still read the target, and MachineDeployment UPDATED read 1 of 3. One PostgreSQL StatefulSet in a namespace I did not own carried a PDB with minAvailable equal to its replica count. Cluster API had done exactly the right thing all night. Fixing it took forty seconds of yaml and one kubectl apply, and the remaining two workers rolled in 16 minutes. Total cost of the misconfiguration: four hours of nothing, and a change window I had to explain. Now the preflight on every cluster I touch starts with kubectl get pdb across all namespaces, before I look at anything else.

Exam focus for objective 4.6

Objective checkpoint

Update discipline for this estate

My recommendation is narrow and unglamorous. Separate version changes from configuration changes into different windows, always, even when it doubles your number of change requests. A version update and a VM class change applied together produce a roll whose duration you cannot predict and whose failure you cannot attribute, because both stages replace the same nodes for different reasons. Run the version change alone, verify with UPDATED and Machine age, then run the configuration change the following day. On this estate that costs an extra window and buys a failure you can name in under a minute.

Avoid the opposite pattern, which is bundling everything into one manifest edit because it feels efficient. It is efficient right up until a drain stalls and you have three candidate causes and a 45 minute head start on nobody.

Tonight, in your own lab: create a PodDisruptionBudget with minAvailable equal to your replica count, start a VM class change, and watch a roll freeze on purpose. Then fix it and watch it resume without any intervention. Twenty minutes of that teaches objective 4.6 better than any amount of reading. If the underlying mechanics still feel thin, the VKS Series covers cluster lifecycle without the exam framing, and the VCAP-VKS exam guide maps every objective to its Part. Part 22 takes the next step and hands node count decisions to an autoscaler, which is objective 4.7.

References

• Updating VKS Service Clusters, VCF 9.0 Service Administration and Development, Broadcom TechDocs
• Understanding the Rolling Update Model for Clusters, Broadcom TechDocs
• Update a v1beta1 Cluster by Editing the TKr Version, Broadcom TechDocs
• VMware Certified Advanced Professional vSphere Kubernetes Service exam guide, 3V0-24.25

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts