Day-2 on OpenShift is not a lighter version of Day-2 on TKGI. It is a different job, with different failure modes, and in most organisations it ends up belonging to a different person. On TKGI (Tanzu Kubernetes Grid Integrated, formerly Enterprise PKS) the platform team owned a sequence of clicks: import a tile into Tanzu Operations Manager, import a stemcell, press Apply Changes, then let BOSH, the release engineering and VM lifecycle engine underneath TKGI, converge every virtual machine it manages. Nothing moved until a human started that sequence, and everything moved when they did.

OCP inverts that. A cluster carries roughly thirty named ClusterOperators that continuously reconcile their own piece of the platform, and a single object called clusterversion that says which release payload the whole thing should be running. You do not upgrade components. You change one field and then watch a fleet of controllers argue with reality until they win. That is a better model, and it is also a model that fails in ways an Ops Manager operator has never seen, usually at four in the morning, usually because of a PodDisruptionBudget somebody copied over during the migration.

Day-2 Ownership Moves From Ops Manager to Operators

Before any commands, it helps to put the two operating models side by side, because almost every argument I have had with a TKGI team in the first quarter after cutover traces back to one row in this table. Print it, put it in the runbook repository, and hand it to whoever is on call. A note on tooling while you are here: I use oc rather than kubectl throughout this series because oc understands OpenShift only resources such as Route, Project and SecurityContextConstraints, and because oc adm carries the cluster administration verbs that have no kubectl equivalent. Everything a kubectl user knows still works.

Row four deserves a sentence on its own. BOSH resurrected unresponsive VMs by default, so TKGI operators grew used to a platform that quietly fixed itself. OpenShift ships the machinery for that but does not switch it on. Until you write a MachineHealthCheck, a worker that loses its kubelet stays broken and stays in the inventory. I have watched three separate teams assume otherwise for weeks.

Preflight Before Any OpenShift Update

Everything below was run against OCP 4.19.9 moving to 4.20.4, with an oc client at 4.20.4, OADP 1.5 in the openshift-adp namespace, Velero 1.16 still installed on the TKGI 1.18 side, and vSphere CSI as the only storage provisioner. Match your client to the target release, not to the current one, or oc adm will hand you stale advice.

That last command is the single best preflight gate I know. It prints every ClusterOperator that is not Available, or is Progressing, or is Degraded. Empty output means the platform believes it is healthy. Start an update with even one line of output there and you are stacking a new failure on top of an old one, which turns a three hour window into a support case.

Next comes the part that has no TKGI equivalent at all. OpenShift will refuse to offer you the next minor release if your workloads are still calling Kubernetes APIs that the target release removes. You clear that in two moves: find the callers, then acknowledge the removal.

Read the acknowledgement key off your own cluster, every single time. That string encodes the release you are on, the Kubernetes minor whose APIs are going away and the release you are heading for, and it changes on every hop. Copying the key from a blog post, including this one, is how people end up patching a key that does not exist and then wondering why the update still refuses to start. Only once the byUser list is empty, meaning nothing is calling the doomed API any more, do you set it to true.

flowchart TD
  A[Read clusterversion and channel] --> B[List apirequestcounts with removals]
  B --> C[Fix callers, then set admin ack]
  C --> D[Set target channel]
  D --> E{Target recommended}
  E -->|yes| F[Start control plane update]
  E -->|no| G[Read the conditional risk]
  G --> F
  F --> H[MCO drains and reboots workers one by one]
  H --> I[Approve pending operator install plans]
  I --> J[Verify cluster operators and routes]

Cluster Update Procedure, 4.19 to 4.20 Step by Step

Step 1, take an etcd backup and prove it exists

This is not ceremony. Since there is no supported way to move a cluster back to an earlier minor, a snapshot taken minutes before you start is the difference between a bad night and a rebuild. Run the Red Hat supplied script on a control plane host and copy the two files off the node immediately.

Step 2, set the channel and read what the graph offers

Two things to notice. Releases promoted to a stable channel land in the matching eus channel at the same moment, so choosing eus-4.20 over stable-4.20 changes nothing about which builds you can reach today. Its real purpose is the Control Plane Only update, formerly called an EUS to EUS update, which lets you go from one even numbered release to the next while rebooting worker nodes only once. That path only exists between even numbered releases, which is exactly why we are heading to 4.20 rather than sitting on 4.19.

Also watch for a Supported but not recommended updates block. Red Hat publishes conditional updates with a named risk and a reference link when a build is known to misbehave under specific conditions, and declares those risks across all channels at once. Read the risk. If it names a driver, a topology or a workload pattern you do not have, take the update. If it names vSphere CSI, stop and wait.

Step 3, start the control plane update

Once the percentage passes roughly seventy and the message names machine-config, the control plane is done and the Machine Config Operator has taken over. From here the clock belongs to your node count, because the operator cordons, drains, reboots and uncordons one worker at a time by default. Nine workers is nine sequential reboots whether the cluster is busy or idle.

Step 4, watch the pools rather than the percentage

UPDATEDMACHINECOUNT should climb by one every eight to twelve minutes on vSphere. If it holds at the same number for more than twenty minutes with DEGRADED still False, you are almost certainly stuck in a drain, not in a reboot, and the next section explains why.

Operator Updates and Channel Discipline

Cluster updates and Operator updates are separate lifecycles, and that separation is where a migrated estate quietly rots. Every Operator you installed during the migration, OADP most of all, tracks its own channel through a Subscription object. If that subscription is set to automatic approval, an Operator can update itself on a Tuesday afternoon and take your backup path with it. Set every production subscription to manual and treat approvals as change tickets.

A CSV, or ClusterServiceVersion, is the object that describes one installed Operator version. Seeing it sit at Pending after a cluster update is normal with manual approval and is not a fault. Seeing it sit at Pending with no InstallPlan at all is a fault, and it usually means the channel you pinned no longer carries an entry compatible with your new OCP minor. Fix that by moving the subscription to the channel the new catalogue actually offers, not by deleting the Operator, which on OADP would take your DataProtectionApplication and its backup schedules with it.

Node Scaling, MachineSets and Health Checks

Scaling was covered as a build activity in Part 14. As a day-2 activity there are only two extra objects worth writing, and both of them replace behaviour BOSH gave you for free. A ClusterAutoscaler sets fleet wide ceilings, a MachineAutoscaler binds a range to one MachineSet, and the MachineAutoscaler only takes effect once a ClusterAutoscaler exists.

Set maxUnhealthy deliberately. Left unset it defaults to one hundred percent, which means remediation proceeds no matter how much of the pool is unhealthy, and a vSphere storage incident that makes six workers go NotReady at once will cause OpenShift to delete six machines while the datastore is still misbehaving. Forty percent short circuits that: past the threshold the check stops remediating and waits for a human, which on a nine node pool means it will replace at most three at a time. Keep MachineHealthCheck away from control plane machines until you are comfortable, because deleting a control plane machine is an etcd member operation, not a reboot.

Verification, Rollback and Common Failures

A clean finish looks like this, and I check all four before I write the change record closed.

Now the uncomfortable part. Rollback of a completed minor update does not exist on OpenShift. You get three fallbacks and they degrade sharply. Before the Cluster Version Operator begins applying a requested target, oc adm upgrade --clear cancels it cleanly. Once the control plane is moving, pausing the worker MachineConfigPool with oc patch machineconfigpool worker --type merge -p and a paused true field freezes your nodes on the old operating system while you decide, which buys hours rather than minutes. Past that point the only route back is restoring the etcd snapshot from Step 1 across the whole control plane, which is a full disaster recovery procedure and costs far more downtime than the update ever would. Plan forward, not backward.

With that fixed, the second attempt produced timings I now use for planning. They are worth plotting, because the shape tells you where to spend engineering effort and it is not where most people assume.

Half the window is worker reboots you cannot compress by being clever, which is a useful thing to know before somebody asks you to shorten the change window. What you can compress is how often you pay it, and that is a release selection decision rather than an operational one.

Pin New Clusters to Even Releases and Budget for Reboots

Here is where I disagree with what almost every installation walkthrough implies. We installed 4.19 because it was current on the day we built the cluster, and taking the newest release felt like the responsible choice. It was not. OpenShift ships a minor roughly every four months, gives every release eighteen months of Maintenance Support, and reserves Extended Update Support entirely for even numbered releases, where add on terms stretch a single release to twenty four, thirty six or forty eight months. Landing on 4.19 meant we had bought an eighteen month clock with no extension available, and our first day-2 task after a migration that took nine months was another migration of sorts. Build new clusters on the current even numbered release even if that means installing one minor behind, then ride the Control Plane Only path from even to even and reboot your workers once per hop instead of twice.

A clean day-2 posture, three months after cutover, reads like this. Every ClusterOperator Available and neither Progressing nor Degraded. Both MachineConfigPools Updated with zero degraded machines. Every production Subscription set to manual approval with a named owner. A MachineHealthCheck on each worker MachineSet with maxUnhealthy well below one hundred percent. An etcd backup taken and copied off cluster within the last twenty four hours. Channel set to the eus stream of the current even release. Zero apirequestcounts reporting a removedInRelease value. If any one of those is false, fix it before you plan the next update, because updates surface debt rather than creating it.

Worth one clause for completeness: if you are reading this while still choosing a destination and staying inside the VMware ecosystem matters more to you than the Operator model, the TKGI to VKS guide covers that landing place instead. And if OpenShift is now your platform and GPU workloads are on the roadmap, the Red Hat Gen AI guide picks up OpenShift AI and model serving from here. Part 25 takes on observability, backup and disaster recovery, which is the other half of owning this platform.

On Monday, run one command against your own cluster: oc get pdb -A. Every PodDisruptionBudget whose minAvailable equals the replica count of its Deployment is a node drain that will hang during your next update. Fix those now, while nothing is on fire.

References

• Red Hat, Performing a cluster update, OpenShift Container Platform documentation
• Red Hat, OpenShift Container Platform Life Cycle policy
• Red Hat, Operators administrator tasks, update channels and approval strategy
• Red Hat, Deploying machine health checks, Machine management documentation
• Red Hat Developer, A guide to the oc adm upgrade recommend command

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts