A production cutover I ran last year came down to one number I nearly left alone, a DNS record with a time to live of 3600 seconds. That single hour of cached lookups turned a clean traffic shift into two hours of split traffic, writes landing on both platforms, and a reconciliation nobody enjoyed. Cutover is the moment a migration stops being a rehearsal, so the plan has to treat traffic, validation and rollback as one connected move rather than three hopeful steps.

Cutover approach and current state

Last part ran the full playbook against the dev cluster as a pilot, timed every phase, and scored the exit criteria to a go. Prod workloads already sit on VKS from the staging rehearsal, restored and validated, waiting behind their own address. This part flips production traffic from the TKGI cluster to the VKS cluster, proves the new platform is serving real users, and keeps a clean path back if it is not.

Here is where the usual advice falls down. Most cutover guides say change the DNS A record and you are done, as if one edit moves every client at once. Resolvers and clients cache the old answer for the length of the TTL, and a few ignore low TTLs entirely, so a bare DNS change gives you a slow, uneven shift you cannot reverse quickly. A controlled cutover happens at a layer you own, a GSLB or a reverse proxy sitting in front of both platforms, with DNS as the coarse outer layer whose TTL you lowered days ago.

Three cutover shapes are worth weighing, and the right one depends on how much reversibility you need. Pick the shape before the window opens, because deciding under a rising error rate is how a shift becomes an outage.

For a production estate with users and data, the weighted GSLB shift is the pick, because it turns a binary flip into a dial you can turn back. Avoid the bare big bang DNS flip for anything with an error budget, since it commits every client on a timescale you do not control and offers no fast reverse. A blue green reverse proxy is even cleaner where you can put one proxy in front of both platform load balancers, though two separate estates, TKGI on NSX-T and VKS on Avi, often make a GSLB the practical steering layer between them.

Preflight and change-freeze checks

Prove five things before the window opens: the DNS TTL was lowered a week ago and has propagated, the target VKS cluster is healthy and serving on its own address, a fresh final backup of the production namespaces exists, writes on the source can be frozen at the top of the window, and a named owner holds the rollback decision. A preflight that skips the TTL check is exactly how my worst cutover started, so it goes first.

Here is the artifact worth keeping, a T minus timeline that turns cutover from a scramble into a schedule. Each row names when, what, and the proof it actually happened, so nobody flips a record on trust. Copy this into your change ticket and check the proof column live.

Traffic shift, DNS and load balancer cutover

With preflight green, the shift runs in four steps. First, freeze writes on the source and take a final delta restore so the target holds the newest data. Second, confirm the VKS pool is healthy at the GSLB so traffic has somewhere safe to land. Third, move 10 percent of traffic and watch the error rate for a few minutes. Fourth, move to 100 percent while leaving the source pool defined at ratio zero, warm and ready for a failback.

Steps three and four both live at the GSLB, where a member ratio decides how much traffic each platform receives. Moving in a 10 percent step means a bad shift touches a tenth of users for a few minutes, not everyone at once, and the reverse is one API call away. Read the controller token from an environment variable, never a hardcoded string, so the credential never lands in a shell history or a runbook.

First attempt at the 10 percent step sent zero traffic to VKS, and the GSLB showed the new pool member DOWN. A health monitor was probing the site root, which answers a 302 redirect to the login page, and the monitor only counted a 200 as healthy. So the balancer decided the target was broken and refused to send it anything.

Production validation and what green looks like

Green is not a pod reporting Running and it is not a 200 on a health path. Green is the error rate holding inside its budget, a real write persisting on the target, and no writes still landing on the source after the shift. Validate all three at the 10 percent step, before you commit the other 90 percent, so a problem shows up while it is still small.

A false green is the trap. An ingress can answer 200 from a default backend that knows nothing about your service, and a read only test passes even when the storage mount is broken for writes. Drive one real order through the path, confirm it landed in the database on the target, then confirm the source log shows zero writes since the shift. That last check catches split traffic, where a client with a stale cache is still writing to TKGI while everyone else writes to VKS.

Watch the error rate shape, not just its peak. In this cutover the 10 percent step nudged 5xx responses to 0.11 percent, a hair over the 0.10 error budget, for a single four minute bucket while connections drained on the old side, then it settled back under budget on its own. A brief overshoot that recovers is normal, a rate that keeps climbing is your signal to shift the weight back.

Rollback and failback to TKGI

Because TKGI was kept warm and untouched, rollback is a traffic move, not a data recovery. Shift the GSLB weight back to the source, confirm it is serving within one TTL, and users land where they started. That is the whole point of leaving the old pool defined at ratio zero rather than tearing it down at cutover, a failback measured in seconds instead of the hours a restore would take.

One part of rollback is genuinely hard, the writes that landed on VKS during the shift. Any order taken on the target while the cutover was live does not exist on the source, so a naive failback loses them. Because the source stayed warm you can reconcile instead of losing data: export the delta rows written to the target, replay them on the source, then tear the target down. Count the rows on both sides before you decide, so the choice to keep or discard is a number and not a guess.

stateDiagram-v2
  [*] --> Rehearsed
  Rehearsed --> Shifting: lower TTL, shift 10 percent
  Shifting --> Validated: error rate under budget
  Validated --> LiveOnVKS: shift to 100 percent
  Shifting --> RollingBack: budget breached
  RollingBack --> BackOnTKGI: weight back to source
  LiveOnVKS --> [*]
  BackOnTKGI --> [*]

Below is the failure lookup I keep open during a cutover. Each row is a symptom I have actually seen, its cause, and the fix, so a stall becomes a page turn rather than a debugging session with users waiting.

Cutover recommendation and next move

My worst production cutover skipped one line of the timeline above. Someone left the DNS TTL at 3600 because lowering it a week early felt like premature risk. We flipped the A record at the top of the window, and forty percent of traffic was still hitting TKGI two hours later while caches slowly expired. Orders split across both platforms, and I spent the next morning reconciling about 1900 rows by hand. Every cutover I have run since steers at a GSLB and lowers the TTL seven days out, no exceptions.

For the traffic layer, my pick is a weighted GSLB shift with the TTL lowered a week ahead, because it gives you a dial and a fast reverse. Avoid the bare DNS flip for anything carrying real users or real writes, since it hands control of the timeline to every resolver on the internet. For the safety layer, keep TKGI warm and untouched through the full rollback window, so a bad cutover is a weight change and not a restore from backup.

One decision splits teams, how long to keep the source warm after a clean cutover. A day feels tidy, but the failures that force a rollback, a slow memory leak on the new platform or a batch job that only runs nightly, often surface after the first full business cycle. Hold the source at ratio zero for at least one business cycle past cutover, three to five business days in the running example, then decommission with the confidence that nothing hidden is waiting to bite.

Two questions come up on every cutover call. Can we cut over with a plain DNS change if we have no GSLB? You can, but lower the TTL well ahead and accept a slow, hard to reverse shift, or stand up a simple reverse proxy in front of both platforms for the day so you have a control point. Do we take a backup even though the pilot passed? Always, a pilot proves the procedure while a fresh backup protects the specific data you are about to move, and the two are not interchangeable.

For the mechanics behind this move, the ingress and DNS cutover part sets up the addresses this shift steers, and the pilot part is the rehearsal this production run repeats. The TKGI to VKS guide holds the full map, and the series hub links the related VKS and VCF 9 work.

On Monday, open your DNS zone and check the TTL on the records you will cut over. If any still read 3600, lower them to 60 today, because that one edit a week early is the difference between a cutover you control and a cutover that controls you. Next part takes the migration to its end, decommissioning TKGI and Ops Manager once the warm window closes.

References

VMware Cloud Foundation blog, Mastering Application Migration to VKS
Velero docs, cluster migration case
Velero on GitHub, backup and migrate Kubernetes applications

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts