Our first production cutover rehearsal produced this, and both clusters were completely healthy at the time:
Eleven seconds of hard 503, no pod restarts, no failed probes, no NSX alarm. What happened was that I patched the two Route backend weights with two separate commands, and for those eleven seconds both weights were zero. Red Hat documents exactly this behaviour: if every backend weight on a Route is zero, the router treats the Route as having no backends and returns a 503. A cutover is full of small mechanical facts like that one, and production night is a bad time to discover them.
1. Do not use DNS as the traffic shift control. Move DNS to the OpenShift ingress VIP days early, park the Route on a proxy that still forwards to TKGI, and make Route weight the only lever you touch on the night.
2. Headline command, and it must be one atomic patch, never two:
oc -n prod-web patch route/shop --type=json -p '[{"op":"replace","path":"/spec/to/weight","value":50},{"op":"replace","path":"/spec/alternateBackends/0/weight","value":50}]'3. Route weights run 0 to 256 with a default of 100, and a Route accepts at most three alternate backends. Share is weight divided by the sum of weights, so 90 and 10 is genuinely 90 percent.
4. Pre-seed bulk volume data 24 hours ahead. Our pilot spent 185 minutes on restore; the production window spent 55, because 214 GiB had already landed.
5. Rollback is a patch that completes in under two seconds. Keep it armed through a 60 minute soak at 100 percent, and rehearse it before you need it.
Cutover Window Sizing From Pilot Timings
Part 22 finished with a number that nobody enjoyed reading: a six hour pilot plan that took 9 hours and 40 minutes, or 580 minutes of wall clock. That number is not the production window. It is the raw material for building one. Three of the pilot phases were one time costs that never repeat, one phase can be moved out of the window entirely, and one phase does not exist in a pilot at all.
Work through it line by line rather than applying a contingency multiplier to the total. A multiplier hides which phase is actually risky, and it produces a window that a change board will reject on principle.
One more discipline before the table. Size the window against the phase you cannot interrupt, which is the stretch between freezing writes and reconciling row counts. Everything before that is reversible by doing nothing, and everything after it is reversible by a Route patch. In our plan that uninterruptible stretch is 80 minutes of the 270, and it is the only number the business genuinely needs to hear, because it is the only period during which the application is unavailable for writes.
| Phase | Pilot actual | Production plan | Why the difference |
|---|---|---|---|
| Preflight | 55 min | 40 min | Scripted after the pilot, still run by a human reading output |
| Bulk volume pre-seed | not attempted | 0 min in window | 214 GiB restored at T minus 24 hours, outside the change window |
| Freeze and final backup | 72 min | 25 min | Only the 24 hour delta moves, not the full dataset |
| Restore and reconcile | 185 min | 55 min | Persistent volumes already populated by the pre-seed |
| Projects and identity | 95 min | 0 min | One time cost, paid in Part 15 and proved in the pilot |
| Weight ladder and soak | not applicable | 120 min | New work in production, a pilot has no live traffic to shift |
| Verification and sign off | 173 min | 30 min | Pilot scorecard turned into a script that exits nonzero |
| Total | 580 min | 270 min | Ask for 4 hours 30 minutes, publish 6 hours to the business |
Note what the pre-seed does to the shape of the night. Moving 214 GiB of PostgreSQL data out of the change window converted the single scariest phase into a delta copy. It also means the OpenShift persistent volumes are already bound, already attached to nodes, and already proved by vSphere CSI (Container Storage Interface, the driver that provisions vSphere disks as Kubernetes volumes) before anyone is awake to care.
Preflight, Freeze Gates and Version Baseline
State your versions before your commands, because a cutover runbook that does not name its versions is a runbook nobody can reproduce. Everything below was executed against OCP 4.19.6 with the matching oc client, OADP 1.5.0 which packages Velero 1.16, and Velero CLI 1.16.0 driving the source TKGI 1.18 cluster. Red Hat supports one OADP version per OpenShift version, so check the pairing before you upgrade either one mid migration.
Two of those checks earn their place because of what they prevent. A 60 second TTL on the A record means a bad DNS decision costs you a minute of stale answers instead of an hour. Two ready router pods means that shifting traffic onto OpenShift does not simultaneously concentrate all of it onto one HAProxy instance, which is how a cutover turns into a capacity incident.
Concretely, that means standing up a small reverse proxy Deployment in the target project whose only job is to forward to the existing NSX-T VIP, exposing it as a Service called shop-legacy-proxy, and pointing the Route at it. Customers reach OpenShift ingress and get served by TKGI. Nothing about the application has changed, DNS has already settled, and every subsequent move is a patch against one Route object.
stateDiagram-v2 [*] --> Preflight Preflight --> Frozen: gates green Frozen --> Restored: delta backup and restore done Restored --> Shadow: verified on shadow hostname Shadow --> Shifting: weight ladder starts Shifting --> Soaking: 100 percent on OpenShift Soaking --> Landed: 60 minute soak clean Shifting --> RolledBack: abort trigger fires Restored --> RolledBack: row counts disagree RolledBack --> Frozen: fix, then retry Landed --> [*]
Cutover Procedure, Step by Step
Step 1, Freeze Writes and Take the Delta Backup on TKGI
Freeze at the edge, not in the database. Scaling the web tier to zero stops new writes while leaving the database live for the backup and for the reconciliation query. Capture a business number before the backup runs and hold it, because that number is your only honest proof that no orders were lost.
Step 2, Restore on OpenShift and Reconcile
This is where a migration that has been polite for twenty two Parts finally bites. Our production database image runs as a fixed UID, which TKGI permitted under its PodSecurityPolicy posture and which restricted-v2, the default Security Context Constraint (SCC) on OpenShift, rejects outright. We had fixed this in staging. Nobody had applied the fix to the production project.
Matching row counts is the gate. If they disagree, you are in the RolledBack state on the diagram above, and the correct move is to scale the TKGI web tier back up, release the freeze, and go home. Prefer oc over kubectl on the OpenShift side throughout, because subcommands like oc adm policy and oc get route have no kubectl equivalent, and mixing the two clients in one runbook is how people run the right command against the wrong cluster.
Step 3, Prove the Target on a Shadow Hostname
Before any customer traffic touches the new backend, exercise it through a second Route on a hostname only you know. Same ingress controller, same TLS termination, same OVN-Kubernetes data path, zero blast radius. Run the application team’s smoke suite against it, not a health endpoint, because a health endpoint proves that a process is alive and nothing more.
That failing line is worth more than the three passing ones. Sticky sessions on TKGI were held by the NSX-T load balancer; OVN-Kubernetes and the OpenShift router do not reproduce that behaviour by default. We caught it on a shadow hostname at 01:15 with nobody watching, instead of at 50 percent traffic with everybody watching.
Traffic Shift With Weighted Routes, and DNS Last
A Route can carry a primary backend plus up to three alternate backends, each with a weight between 0 and 256, defaulting to 100. With roundrobin scheduling the share a backend receives is its weight divided by the sum of all weights, so a 90 and 10 pair really is ninety percent and ten percent rather than an approximation. That arithmetic is the whole traffic shift mechanism, and it lives entirely inside one object you can patch and unpatch.
Set the balance annotation to roundrobin explicitly rather than accepting whatever the ingress controller defaults to. With leastconn, which is a perfectly reasonable default for a steady state application, the weights still apply but connection counts skew the effective share in ways that make a ladder step hard to reason about at three in the morning. During a cutover you want the share to be arithmetic, verifiable against a request count, and boring. Switch back afterwards if leastconn suits the workload better in normal operation.
Now the ladder. This table is the artifact worth keeping from this Part: five steps, each with the exact weights, a dwell time long enough for a metric to move, and a named abort trigger that anyone in the bridge call can read out loud. An abort trigger you have to interpret is not an abort trigger.
| Step | Legacy weight | OpenShift weight | Dwell | Abort trigger |
|---|---|---|---|---|
| 0 baseline | 100 | 0 | 15 min | Any 5xx above the recorded baseline of 0.02 percent |
| 1 | 90 | 10 | 10 min | 5xx rate above 0.20 percent over 5 minutes |
| 2 | 50 | 50 | 20 min | p95 latency above 900 ms, or any failed order write |
| 3 | 10 | 90 | 15 min | Any failed order write, no tolerance |
| 4 | 0 | 100 | 60 min soak | Rollback stays armed, TKGI stays running and warm |
| 5 land it | removed | primary, 100 | next day | Delete alternateBackends and the proxy after 24 clean hours |
Every step is a single JSON patch touching both weights at once. Splitting it into two commands is what produced the 503 at the top of this page, and it is the most common self inflicted wound I see on cutover night.
Verification, Rollback and Common Failures
Our first attempt did not reach step 4. Here is what both attempts looked like plotted against the ladder, which is the single most useful picture to put in front of a change board when you ask for a second window.
Field Note, a Cutover Reversed at 03:40
Attempt 1 was clean through baseline and clean at 10 percent. At 03:18 I patched to 50 and 50. For eighteen minutes the 5xx rate sat at 0.07 percent and I let myself relax. Then it climbed to 1.90 percent over roughly four minutes, concentrated entirely in checkout. Cause: connection pool exhaustion on the migrated PostgreSQL pod, which had inherited a max_connections value tuned for a TKGI node with more memory than the OpenShift worker it landed on. At 03:40, after 22 minutes at 50 percent, I reversed. That patch took 1.4 seconds and the error rate was back to baseline before I had finished typing the message to the bridge call.
Total customer impact: 22 minutes at a raised error rate on half of traffic, roughly 3,100 failed requests. Total cost of the reversal: one 90 minute change window burned and a fortnight of schedule. What made that survivable was not skill. It was that TKGI never went away, and the only thing standing between us and a healthy platform was an integer in a Route. Attempt 2, nine days later with the pool sized correctly and the workers resized, landed with a peak of 0.09 percent and no reversal.
Verdict on dwell times: pick 20 minutes at the 50 percent step, not 10. Our problem took 18 minutes to surface, and a 10 minute dwell would have shipped it straight to 90 percent. Avoid the tempting shortcut of jumping baseline directly to 50 because the pilot went well. A pilot has no connection pools under real load.
| Symptom or error | Cause | Remediation |
|---|---|---|
| 503 on every request, router logs show no backend | Both weights momentarily zero because the patch was split into two commands | Always one atomic JSON patch. A Route with all weights zero returns 503 by design |
| is forbidden: unable to validate against any security context constraint [provider restricted-v2] | Image runs as a fixed UID outside the project range | Bind nonroot-v2 to the service account, or rebuild the image to run as an arbitrary UID |
| Traffic still hitting TKGI 40 minutes after the DNS change | Resolver cached the old A record at the previous 3600 second TTL | Lower TTL 48 hours ahead and stop using DNS as the shift control |
| Phase: PartiallyFailed with warnings only under Cluster | Cluster scoped resources already present from the pre-seed run | Expected on a second restore. Read the warnings, confirm they are pre-existing, do not re-run blindly |
| Sessions dropped once weight reaches 50 | Session affinity was provided by the NSX-T load balancer, not by the application | Move sessions to a shared store before the window, or declare the drop to the business in advance |
| p95 latency doubles at the first ladder step | Cold caches and empty connection pools on the new cluster | Extend the 10 percent dwell rather than skipping it, and check pool sizing against the worker node memory |
Cut Over in Halves and Keep DNS Out of It
Recommendation for this Part, stated plainly: separate the DNS move from the traffic move by several days, make Route weight the only lever anybody touches during the window, and keep TKGI running and warm until you have banked 24 clean hours. Everything else in a cutover plan is negotiable. Those three are not, because together they are what turns a reversal from an incident into a patch.
A clean result looks like this. DNS has pointed at OpenShift ingress for a week and nobody noticed. Row counts match on both sides of the freeze. Every ladder step is recorded with a timestamp, a weight pair, and an error rate. Rollback has been exercised at least once. The soak finished at or below the baseline error rate. Only then do you delete the alternateBackends stanza and the legacy proxy, and only then does the TKGI cluster become a decommissioning candidate rather than a safety net.
Your Monday action: open your production Route definition and write the five row ladder table for one real application, filling in the abort trigger column with metrics you can actually query today. If a row’s abort trigger has no query behind it, you have found the observability gap that Part 25 of this guide exists to close. If you are still weighing whether OpenShift is the right landing place at all, the alternative path is laid out in the TKGI to VKS guide, and it is a legitimate choice for estates staying on VMware Cloud Foundation. Next Part covers what daily life looks like once the traffic is yours: Operators, upgrades and scaling, measured against what BOSH used to do for you.
References
- Route [route.openshift.io/v1], Network APIs, OpenShift Container Platform 4.16, Red Hat Documentation
- Deployments and route based deployment strategies, Building applications, OpenShift Container Platform 4.18, Red Hat Documentation
- Restore Reference, Velero documentation
About The Author
DrJha