Where the migration stands before cutover

My platform lead asked one question before signing off on this window: if it goes wrong at 40 percent, how fast can you put everyone back on TKGI. That question, not the ingress YAML, is what shapes a cutover. Everything up to now was reversible because the source stayed untouched. Part 18 carried the storefront frontend onto the VKS cluster dev-tkg-01, Part 19 moved the webapp database with a checkpoint and a storage class map, and through both the TKGI estate kept serving every real user. This part is where a user request finally lands on VKS instead of TKGI, and the whole craft is making that shift happen in slices small enough that any one slice can be pulled back before anyone files a ticket.

Two terms before the runbook. Ingress is the Kubernetes object that maps a hostname and path to a Service, and it needs an ingress controller to turn that intent into a real load balancer configuration. AKO, the Avi Kubernetes Operator, is that controller here, a pod in the cluster that watches Ingress and Service objects and programs the Avi Load Balancer Service Engines to match. On TKGI the same job was done by the NSX-T load balancer driven by annotations on the Ingress. Those annotations are the first thing that does not survive the move, and pretending they do is the fastest way to a green apply that routes nothing.

The running example for this window is the webapp namespace, a web and api tier in front of the PostgreSQL database Part 19 landed on vks-vsan-default. On TKGI it is reached at shop.example.internal through an NSX-T virtual server at 10.20.30.40. On VKS it will be reached at the same hostname, through an Avi VIP that AKO allocates, and the only thing that decides which one a user hits is a DNS record you control.

Preflight before you touch a DNS record

Four checks earn the right to move traffic, and every one of them is cheaper than a rollback. AKO must be running and watching the ingress class you are about to use. VKS must already have an external VIP for the Service, because a hostname pointed at nothing is worse than no change at all. You must know the current DNS record and its TTL, because that number sets how long a mistake lingers. And the app on VKS must answer correctly when you address it by host header directly, before any user is involved.

What green looks like: the ako pod reports Running, an ingressclass named avi-lb is present, the Service shows an EXTERNAL-IP that is a real address and not Pending, and dig returns the old NSX-T VIP with whatever TTL is set today. On this estate that TTL came back as 300, and 300 is the first thing that has to change, days ahead of the window, not during it.

Expose the app on VKS behind a load balanced ingress

Pick the ingress path before you write a manifest, because the choice decides who owns Layer 7. Foundation Load Balancer and NSX Load Balancer give the Supervisor clean Layer 4 out of the box, but neither does HTTP routing, TLS termination or a web application firewall on its own. Avi with AKO adds Layer 7 at the platform, and because VKS clusters are upstream conformant you can also run an in-cluster controller like Contour or the Gateway API if you would rather own routing inside the cluster. For a hostname that already terminated TLS and did path routing on NSX-T, Avi with AKO is the closest match and the least surprising for the app team.

Table 1. Ingress path decision for the cutover. This is the reference artifact, pick one row per hostname and record it beside the DNS change ticket.

With Avi chosen, the Service and Ingress are short. The Service uses loadBalancerClass so AKO, and not some other controller, claims it, and the Ingress uses ingressClassName avi-lb. This is where the TKGI manifest gets rewritten. On TKGI the same Ingress carried an ncp/ or nsx annotation and no ingressClassName at all, and if you paste that here AKO ignores it and the ADDRESS never fills in.

Apply it, then wait for AKO to program Avi and hand back a VIP. The Ingress ADDRESS is the number you will eventually point DNS at, so do not go further until it is populated and the app answers on it by host header. Address the VIP directly with a Host header, so you are testing the new path with zero users and zero DNS involved.

flowchart TD
  A[app live on VKS, Ingress ADDRESS filled] --> B[lower TTL to 30s, 48h ahead]
  B --> C[shift 10 percent to VKS VIP]
  C --> D{error rate within budget}
  D -- yes --> E[shift 50 percent]
  D -- no --> R[revert weight to TKGI VIP]
  E --> F{p95 latency healthy}
  F -- yes --> G[shift 100 percent]
  F -- no --> R
  G --> H[drain TKGI, watch old VIP hits]
  H --> I[zero hits, retire TKGI ingress]
  R --> A

Lower TTL and shift traffic in weighted steps

The single most common cutover mistake is treating DNS TTL as a same-day setting. A record has whatever TTL it was last served with, and a resolver holds that record until it expires. If your TTL was 300 this morning and you drop it to 30 as you start the window, resolvers that already cached the record at 300 will keep it for up to five more minutes, and some corporate resolvers enforce their own minimum well above yours. Lower the TTL 48 hours ahead, confirm the low value has propagated, and only then start moving weight.

With a low TTL confirmed, shift in steps. On this estate the front was an Avi GSLB service with two pool members, the TKGI VIP and the VKS VIP, and the weight ratio decides the split. Start at 10 percent to VKS, hold, read the error rate and latency, then 50, then 100. The commands below show the DNS side settling to the low TTL and the first weighted step, with the real failure that catches people, an ADDRESS that never filled because the ingress class was wrong.

Verify, roll back, and the failures that bite

Verification during a cutover is not one check at the end, it is a reading at every weight step. At each step confirm three things: the request lands on the VKS VIP for the share you expect, the error rate stays inside budget, and p95 latency on the new path is at least as good as the old. On this estate the VKS VIP answered at 42 milliseconds p95 against 55 on the NSX-T VIP, and the error rate never left the 0.1 percent noise floor, which is what let me move from 50 to 100 without a pause.

Rollback is a weight change, not a redeploy, and that is the point of moving traffic at the balancer rather than in DNS. If any gate fails, set the GSLB ratio back to all TKGI and the next request routes to the old VIP, because the source cluster never stopped serving. DNS does not need to change to roll back, which is exactly why you never leaned on it to move traffic in the first place.

Table 2. Cutover failures and their remediation. Rows one and two hit almost every hostname on the first pass, and row two is the one that turns a 40 minute window into a two hour one.

Field note and verdict

Cut over by draining, not by flipping

A clean result looks like this: AKO programmed an Avi VIP for the shop Ingress, the app answered on that VIP by host header before any user was involved, the DNS TTL sat at 30 seconds a full 48 hours before the window, traffic moved 10 then 50 then 100 percent with error rate inside budget and p95 latency better on the new path, and the old NSX-T virtual server drained to zero active connections while TKGI stayed authoritative the whole time. Hit that and the app is on VKS with a rollback you never needed but always had.

On your own estate on Monday, pick one low-traffic hostname, stand up its Service and Ingress on VKS with loadBalancerClass ako.vmware.com/avi-lb and ingressClassName avi-lb, and prove it green by host header with zero DNS change. Then lower that record TTL to 30 seconds and leave it for two days before you touch weight. For the load balancer options underneath this, the Supervisor Networking and load balancer choices part lays out FLB, NSX-LB and Avi in full. The complete migration map lives on the TKGI to VKS guide, and the related VKS, VCF 9 and NSX series sit on the guides hub. Next part moves secrets, config and the CI/CD pipeline, so the app that now answers on VKS is also built and deployed there.

Questions worth answering

Why not just change the A record to the new VIP in one go?
Because rollback then depends on DNS caches you do not own, which can hold the new record past your TTL. Moving weight at the load balancer takes effect on the next request and reverses just as fast, so DNS only points at a stable front you re-weight behind.

Do my TKGI Ingress manifests carry over to VKS?
Not unchanged. NSX-T annotations mean nothing to AKO, and there is no ingressClassName on the old object. Rewrite each Ingress with ingressClassName avi-lb and drop the old annotations, then re-add only the AKO annotations you actually need.

How far ahead should the TTL come down?
At least 48 hours, because the record is cached at the old value until that value expires, and some resolvers enforce a minimum above yours. Set it low, confirm propagation from a few external resolvers, and only then start the window.

When is it safe to retire the TKGI ingress?
When the old virtual server reports zero active connections and stays there across a full cache lifetime, not the moment DNS shows the new VIP. Stragglers behind slow resolvers keep arriving on the old VIP for a while, so watch the old side, not the new one.

This series covers a production migration. Run the cutover in a change window, lower TTL days ahead, shift traffic in reversible steps, and keep the TKGI side authoritative until the old VIP drains to zero.

References

• Load Balancing in vSphere 9.0+ and VCF 9.0+, choosing FLB, NSX-LB or Avi, VMware Cloud Foundation blog
• Avi Kubernetes Operator for VKS on vSphere Supervisor, Broadcom TechDocs
• Ingress, Kubernetes documentation

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts