Day-2 model and where the migration stands

A platform lead on my team asked one question in the first week after cutover: where did the BOSH director go, and what runs the upgrades now. It is the right question, because Day-2 on VKS is not a smaller version of Day-2 on TKGI, it is a different machine. On TKGI you told BOSH what you wanted and BOSH deployed it, step by step, until it finished. On VKS you write the desired state into a Kubernetes object and a controller reconciles the cluster toward it, continuously, whether or not you are watching.

Last part cut production traffic over to the VKS cluster and kept TKGI warm through the rollback window. That window has now closed cleanly, so this part is about living on VKS: upgrading Kubernetes versions, scaling node pools, and backing out when a change goes wrong. Ops Manager and BOSH do not exist on this platform, and their absence changes how every routine task is done. What used to be a tile click and a long deploy is now a field edit and a watch.

That single shift, from imperative deploy to declarative reconcile, is the thing to internalize before any command makes sense. A reconcile loop is patient and self-correcting, but it also means the platform, not your runbook, decides the order and pace of a change. Your job moves from running the upgrade to describing the end state correctly and reading what the controller is doing.

Lifecycle models compared, BOSH deploy vs declarative reconcile

Both platforms give you managed Kubernetes, but the operating model underneath is opposite in one way that matters. BOSH is imperative at the seams, you run tkgi upgrade-cluster or bosh deploy, a process executes, and when it finishes the change is done. VKS is declarative all the way down, you change a field, a controller notices the drift between desired and actual state, and it closes the gap on its own schedule. That difference decides how you upgrade, how you scale, and above all how you roll back.

Here is the verdict for anyone weighing the two. For steady-state operations the declarative model wins, because a controller that continuously reconciles catches drift you would otherwise fix by hand. Where BOSH still feels safer is rollback, and that gap is real, on VKS a Kubernetes minor version upgrade is one directional and you cannot edit the VKr back to undo it. So the reconcile model buys less operational toil at the cost of a harder undo, which is exactly why Velero backups move from nice to have on TKGI to mandatory on VKS.

flowchart TD
  A[Edit cluster spec, new VKr version] --> B[Cluster controller detects drift]
  B --> C[Roll a new control plane machine]
  C --> D[Drain and replace each worker machine]
  D --> E{Health checks pass}
  E -->|yes| F[Cluster Provisioned at new version]
  E -->|no| G[Pause rollout, fix blocker or restore Velero]
  G --> D

Preflight before a cluster upgrade

Prove three things before you touch a production cluster version: the target VKr is present and marked compatible, the jump is a single minor version, and a fresh Velero backup exists because there is no downgrade to fall back on. Skip the compatibility check and you learn the answer mid rollout, with half your nodes on the new version and half on the old.

Kubernetes does not allow skipping a minor version, and the admission webhook enforces it. A jump from 1.30 straight to 1.32 is rejected before a single node moves, which is the safe failure. A worse version of this mistake is assuming the platform will let you catch up in one hop after you have fallen behind on patching, so stay within one minor of the latest compatible VKr and you never meet this wall.

Rolling upgrade of a VKS cluster, step by step

An upgrade is one edit and a watch. First, patch the cluster topology version to the compatible VKr you picked. Second, watch the machines roll, VKS provisions a new control plane node, joins it, retires an old one, then walks the worker MachineDeployments the same way, one machine at a time. Third, confirm every node reports the new version and the cluster phase returns to Provisioned. No tile, no director, no separate upgrade binary, the controller does the work.

On my first production upgrade this rolled cleanly through the control plane and then stopped dead on worker two for 40 minutes. A PodDisruptionBudget on the payments app was set to allow zero disruptions, so the drain could never evict the last pod and the machine sat in Deleting. Relaxing the budget to allow one eviction let the rollout finish in another eight minutes. A budget that permits no disruption and a rolling upgrade that must evict every pod cannot both win, and the controller will wait rather than break your availability promise.

Keep the mapping below within reach for the first month on VKS. It is the reference artifact for this part, the muscle-memory translation from the BOSH and TKGI commands your team knows by heart to the kubectl and VCF CLI equivalents that replace them. Pin it next to your terminal, because the wrong reflex here wastes an afternoon.

Numbers help set expectations for the first upgrade. On the running example, a cluster with three control plane and six worker nodes, a clean VKS rolling upgrade ran end to end in 54 minutes, roughly eight to nine minutes per machine as each drained and rejoined. The equivalent tkgi upgrade-cluster on the old platform took about 62 minutes for the same shape. The version with the PodDisruptionBudget stall took 94, and that gap is the cost of a preflight you skipped.

Scaling nodes, manual and autoscaler

Scaling is where the declarative model trips up BOSH veterans most, because the obvious command does nothing. Node count on VKS lives in the MachineDeployment replicas field, not in any kubectl scale of a workload. To add workers you scale the MachineDeployment, and the same controller that ran the upgrade provisions the new nodes and joins them.

For elastic workloads, install the Cluster Autoscaler as a standard package and let node count follow demand. Here is the catch that cost me an afternoon: the autoscaler reads its bounds from min and max annotations on the MachineDeployment object, and if you put them on the Cluster or a MachinePool it silently ignores the group. The pod stays healthy, it just never scales, which is the worst kind of failure because nothing looks broken.

Rollback and failure remediation

Rollback on VKS is not a version downgrade, and pretending otherwise is how you lose data. A Kubernetes minor upgrade cannot be reversed by editing the VKr back, the controller refuses it. So rollback has two real shapes. For a rollout still in progress, pause it and clear the blocker, a strict PodDisruptionBudget or a stuck node, so it completes forward. For a cluster that upgraded but then misbehaves, restore the workloads from the Velero backup you took in preflight onto a cluster running the known-good version. That is the whole reason the backup is mandatory and not a formality.

Scaling, by contrast, is fully reversible, so treat it differently. If a scale up causes contention, scale the MachineDeployment back down and the extra machines are drained and removed. The asymmetry is worth holding in your head: version changes are one way and need a backup, capacity changes are two way and do not.

Below is the failure lookup I keep open during Day-2 work. Each row is a symptom I have actually hit, its cause, and the fix, so a stall becomes a page turn instead of a debugging session.

Day-2 operating verdict and next move

My worst Day-2 lesson on VKS was not the upgrade stall, it was assuming an upgrade would cascade. I upgraded the Supervisor to a new build expecting the three guest clusters to follow, the way an Ops Manager tile upgrade pulls the whole foundation along. Nothing happened to the clusters. They sat on the old Kubernetes version for two more weeks until I learned that Supervisor and VKS clusters upgrade independently, and that you opt each cluster in by patching its VKr version. On TKGI the platform upgrade and the cluster upgrade were coupled through Ops Manager. On VKS they are deliberately decoupled, so you can patch the Supervisor for a security fix without forcing a Kubernetes version bump on every workload the same night. Once I read that as a feature and not a bug, our upgrade cadence got calmer.

For the lifecycle model, my pick is to lean fully into declarative operations, manage clusters as Kubernetes objects, keep them in version control, and let the controller reconcile. Avoid the instinct to script imperative upgrade runbooks the way you did around tkgi upgrade-cluster, because a controller that already reconciles does not need a babysitter, it needs correct desired state. For rollback, treat every minor version upgrade as one directional and take a Velero backup first, no exceptions, because the downgrade escape hatch BOSH gave you is gone.

Two questions land on every Day-2 handover. Do we still need to schedule upgrade windows now that it is a rolling replace? Yes, a rolling upgrade is graceful but it still cycles every node and evicts every pod, so run it in a change window and check your PodDisruptionBudgets first. Can we let the autoscaler manage production without supervision? Set a sane max so a runaway workload cannot provision the datastore into the ground, and alert on the cluster hitting that max, because a group pinned at its ceiling is capacity you planned to have and do not.

For the backup mechanics this rollback depends on, the Velero toolchain part set up the source and target backups, and the production cutover part is the move that got you here. The TKGI to VKS guide holds the full map, and the series hub links the related VKS and VCF 9 series.

On Monday, pick one non-production VKS cluster and run kubectl get vkr against it. If a newer compatible release is listed, take a Velero backup and patch the topology version by one minor step, then watch the machines roll. Doing it once on a cluster nobody depends on turns the production upgrade from a leap into a repeat. Next part covers observability, backup and disaster recovery on VKS, the safety net under everything you just learned to change.

References

Broadcom TechDocs, Understanding the Rolling Update Model for VKS Clusters
Broadcom TechDocs, Updating VKS Service Clusters
Broadcom TechDocs, Autoscaling VKS Clusters

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts