Nothing was wrong with that cluster, and nothing was wrong with the three variables the webhook named. Every one of them had been valid for the eighteen months the cluster had been running. I had opened the manifest to add one storage class and, while I was in there, retyped spec.topology.class to a newer ClusterClass because it looked untidy sitting a version behind. That single extra line turned a two minute edit into a rejected patch and a list of complaints about fields I had not touched. Objective 4.14 lives precisely in that gap between a manifest that is syntactically fine and an edit the platform will actually accept.

Three kinds of edit and the preflight that gates them

A VKS cluster manifest looks like one document, so operators treat it like one document. It is not. Under spec.topology there are four regions with completely different behaviour when you change them. Replica counts under controlPlane and workers scale in place by adding or removing machines. Variables under spec.topology.variables split into ones that render into node configuration, which forces every machine to be replaced, and ones that only update lists the cluster reads at runtime, which change nothing about the machines at all. And spec.topology.class plus spec.topology.version sit at the top of the chain, where an edit either triggers a whole cluster upgrade or gets thrown out by the webhook.

Before touching any of it, take a copy you can diff against. This is the one preflight step that separates a recoverable evening from a bad one, because a Cluster object has no undo and no revision history. Two more checks matter on this estate. Confirm which ClusterClass the cluster currently sits on, because variable names moved between v3.1.0 and v3.2.0 and a manifest written against the old schema will be rejected wholesale on the new one. Then confirm which VM classes and storage policies are actually bound to the vSphere Namespace, because an edit naming a class the namespace does not have will be accepted by the webhook and then strand your new machines in Provisioning forever. That is the Supervisor reaching up and deciding the outcome of a workload cluster edit, and it happens in almost every failure in this Part.

That last command is the one candidates skip. If guaranteed-large were missing from the binding list, an otherwise perfect vmClass edit would sail past validation and then sit still, and you would spend twenty minutes reading Cluster API logs for a problem that lives in vCenter. Use the table below as the reference artifact for this Part. It is the lookup I keep open during any live cluster edit, and it answers the only question that matters in the moment: what does this line cost.

Numbered run, four edits and one rejection

Last Part we proved the payments namespace could be backed up and restored with Velero. This Part we modify the same cluster in place, four times, and deliberately walk into the rejection that catches people. Work from the Supervisor context throughout. Cluster objects live on the Supervisor, not inside the workload cluster, so a kubectl session pointed at the workload cluster cannot see them at all. That is the single most common reason a candidate reports that kubectl get cluster returns nothing.

Step 1, resize every node with one variable

Vertical scaling on VKS is a variable edit, not a vCenter operation. Change vmClass and the controller provisions replacement machines on the new class and retires the old ones one at a time. Rehearse it with a client side patch first so you can read the resulting object without committing anything.

A new machine in Provisioning within a minute is what green looks like here. If that machine sits in Provisioning past five minutes, stop reading Cluster API logs and go look at the vSphere Namespace, because the usual cause is a VM class that was never bound to it. Part 17 covers the binding, and the pattern generalises: a workload cluster edit fails because a Supervisor object does not permit it.

Step 2, offer a second storage class without touching a node

Here is the edit people get wrong twice. Older VKS releases exposed a flat storageClasses variable. From VKS 3.2.0 the schema became hierarchical and that list moved under vsphereOptions. Broadcom documents the mapping, and using the old name on a current ClusterClass produces exactly the variable is not defined message from the opening block. Both forms are shown so you can recognise either one on sight.

No machine moved. That is the correct outcome and it is worth internalising, because a candidate who assumes every variable edit is expensive will refuse to make a two minute change during business hours. Note also that the storage policy must already be assigned to the vSphere Namespace before it can appear here at all, which is the same Supervisor dependency Part 26 worked through.

Step 3, the rejection, and the correct path to a newer ClusterClass

Now reproduce the failure from the top of this Part on purpose. Change spec.topology.class to a newer builtin-generic version and save. Broadcom is explicit that this is not how rebasing works and that you should not manually edit the ClusterClass version, but the manifest gives you no hint of that, and the error message points at variables rather than at the field you actually changed.

Two details decide whether this works. Clusters created before VKS 3.2.0 carry the skip-auto-cc-rebase annotation automatically after the service is upgraded, so the removal above is mandatory on them and a no operation on newer clusters. And removal only takes effect when an upgrade starts, so removing the annotation on its own changes nothing, and removing it during an upgrade already in flight is too late. Confirm the exact VKr string with kubectl get vkr on the Supervisor before you type it, because a version that does not exist is accepted into spec and then quietly never reconciles.

Step 4, force a redeployment when an accepted edit goes nowhere

On VKS 3.3.0, 3.3.1 and 3.3.2 with builtin-generic-v3.3.x, node configuration is held in a Secret that is referenced by a KubeadmConfig per machine. When you change osConfiguration.trust or systemProxy or resourceConfiguration.systemReserved, the service rewrites that Secret in place and leaves the name alone, so nothing downstream notices and no machine is replaced. kubectl reports success. Your nodes keep the old configuration. Broadcom documents the workaround and states that automatic redeployment after these changes arrives in 3.3.3.

UPDATED lagging REPLICAS is the rollout in progress. That labels block stays in the manifest until you remove it, and changing the value again queues another full redeployment, which is why I date stamp the value rather than leaving it as the word manual. Two engineers reaching for the same trick a week apart with the same value produce no rollout at all and a long argument about whether the command worked.

Verification, rollback and a failure lookup

flowchart TD
  A[Open cluster manifest on Supervisor] --> B{Which field changed}
  B -->|replicas| C[Machines added or removed]
  B -->|vmClass or osConfiguration| D[Rolling replacement of all machines]
  B -->|availableStorageClasses| E[No machine churn, list refreshed]
  B -->|version| F[VKr upgrade plus auto rebase]
  B -->|class by hand| G[Webhook rejects the patch]
  D --> H[Watch kcp md ma until UPDATED equals REPLICAS]
  C --> H
  F --> H
  E --> I[Read storageclasses inside the cluster]
  G --> J[Revert class, change version instead]
  H --> K[Compare against the saved before file]
  I --> K

Rollback deserves plain talk. A Cluster object has no rollback verb. What you have is the before file you saved in preflight and the fact that most edits are symmetric: put the old vmClass back and the cluster rolls again to the original size, at the same cost in time. Two edits are not symmetric. A version bump cannot be reversed, because VKS does not downgrade Kubernetes releases, so treat spec.topology.version as a one way door and rehearse it on a throwaway cluster first. A ClusterClass rebase inherits that same one way property, since it only happens as part of a version change. Reducing control plane replicas from three to one is technically reversible but destroys quorum on the way through, so it is not something to do casually on a cluster carrying work.

Exam focus for objective 4.14

Objective checkpoint

Editing habits worth standardising

My war story is the one at the top, and it cost more than pride. That rejected patch happened at 22:10 inside a 90 minute Tuesday window. I spent 25 minutes convinced the storage class was the problem, because the storage class was what I had come to change and the error named storage first. Only when I diffed against a manifest from the previous week did the class line show up. Reverting it and reapplying just the storage variable took under two minutes. Twenty five minutes lost to an edit I made for tidiness, on a field I had no business touching.

So, three habits. Save the object to a file before every edit, without exception, because diff is the only rollback you have. Change one thing per edit, since the webhook error names symptoms rather than causes and a single field change makes the cause obvious by elimination. And prefer kubectl patch with an explicit JSON pointer over kubectl edit for anything scripted or repeated, because an interactive editor invites the exact opportunistic tidying that started this Part. My verdict on the popular study advice for this objective: memorising the full variable schema is the wrong investment. Schema names have already moved once, between VKS 3.1.x and 3.2.0, and they will move again. What transfers is the blast radius table above and the rule that the Supervisor owns the template while you own the intent. Tonight, take one lab cluster, save its manifest, change vmClass, and time the rollout yourself. A number you measured is worth more under exam pressure than a number you read. Next Part opens Section 5 and moves from making changes to diagnosing them, starting with Supervisor and VKS provisioning failures.

References

• Variable conversion for vSphere Kubernetes Service clusters, Broadcom Knowledge article 404143
• Cluster and variable validation failed when changing the ClusterClass by hand, Broadcom Knowledge article 414721
• Forcing a manual rolling redeployment of nodes on ClusterClass v3.3.x, Broadcom Knowledge article 391589
• VMware Cloud Foundation VKS Administrator exam guide, 3V0-24.25

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts