1. Your headline command is oc create route edge web –service=web –hostname=shop.example.com –cert=tls.crt –key=tls.key –port=8080.
2. Author Routes yourself. If you let the ingress to route controller do it, OpenShift names the generated Route after your Ingress plus a random suffix, and a name nobody can predict is a name nobody can pin a certificate rotation or a monitoring check to.
3. A Route carries one primary backend plus up to three alternates, with weights between 0 and 256. That is your in cluster blue green. It cannot send a single request to a Service that still lives on TKGI, so cross platform weight shifting happens at the external load balancer.
4. DNS is an announcement, not a traffic control. Lower the TTL 48 hours ahead, shift weight at the load balancer, and change the record only once the TKGI pool has gone quiet. On my run 1.9 percent of requests still reached the old pool 41 minutes after the record changed.
5. Check the route timeout before you cut, not after. Requests that ran 46 seconds behind the NSX-T virtual server started returning 504 at exactly 30 seconds on OpenShift.
A DNS change is not a cutover. It is an announcement, and every resolver between your users and your zone file is free to ignore it for as long as it likes. That is the blunt version, and it is why this Part spends most of its length inside a load balancer configuration and only a few paragraphs on a zone edit.
Parts 18 and 19 got wave1-web and wave2-data running on OpenShift, verified by row counts rather than pod status. Nothing outside the cluster is talking to them yet. Every real user request still lands on an NSX-T virtual server that NCP, the NSX Container Plugin that turned TKGI Ingress objects into load balancer configuration, built months ago and nobody has touched since. This Part moves that traffic. Two objects change shape on the way: a Kubernetes Ingress becomes an OpenShift Route, which is Red Hat’s own external access object and predates Ingress by several years, and an NSX-T pool member becomes an OpenShift ingress router endpoint.
Preflight, Route Inventory and DNS Facts to Collect
Preflight for a cutover is not a checklist of cluster health. Your cluster is fine. Preflight here is a list of facts about things you do not own: resolver behaviour, certificate expiry, load balancer idle timeouts, and whichever team wrote the firewall rule that lets the NSX-T VIP reach your application and has never heard of your new one.
Step 1. Inventory every Ingress and Service on TKGI
On the OpenShift side I use oc rather than kubectl, because it understands Routes, Projects and Security Context Constraints natively and this Part is mostly Routes. On TKGI there is no oc, so the source side stays on kubectl.
That last row is the one people miss. pg-direct is a raw PostgreSQL port that NCP published on an NSX-T virtual server. OpenShift’s default ingress controller carries ports 80 and 443 and nothing else, so no Route exists that can replace it. It needs either a Service of type LoadBalancer backed by the MetalLB Operator, or a decision that database clients move inside the cluster and stop crossing the network at all. Find these on day one, because each is a separate conversation with a separate team.
Step 2. Record DNS facts, not a DNS diagram
Ask for the authoritative TTL and then measure it yourself, because the two disagree more often than anyone expects. A record that a wiki says has a 300 second TTL frequently turns out to be 3600 in the zone, and a corporate forwarder can hold it far longer than either number.
Lower the TTL to 60 seconds at least 48 hours before the window. Not 24. A resolver that picked up the record at 3600 seconds one minute before your change still holds the old value for another 59 minutes, and any cache with a longer minimum TTL configured locally holds it for longer still. Forty eight hours costs you nothing except a slightly chattier zone and it removes an entire category of cutover night surprise.
Ingress Objects, Generated Routes and TLS Termination
OpenShift will accept your Ingress objects. When you create one, a controller writes a managed Route to satisfy it, and deleting the Ingress deletes the Route. That sounds like a free migration and it is a trap for three reasons. Generated Routes are named after the Ingress plus a random suffix, so web becomes something like web-gnztq and your automation cannot address it. Ingress has no field for re-encryption or passthrough, so anything beyond plain edge termination arrives through an annotation you have to remember. Worst of all, when the controller declines to convert an Ingress it says nothing at all: no event, no status condition, no log line you will find without asking Red Hat support what happened.
Verdict: write Routes. Keep Ingress objects only for third party charts that hardcode them and cannot be patched, and even then pin the hostname explicitly. Restoring TKGI Ingress manifests with OADP and letting them auto convert is the option to avoid, because a restored Ingress claims your live production hostname the instant it lands.
Step 3. Create the Route on a test hostname first
Never point the first Route at the production hostname. Build it on a name only you know, prove it serves, then add the real name as a second Route once the load balancer is ready. Certificates come from files that live in a mode 0600 directory and never enter git. If you script this, read the key path from an environment variable such as $TLS_KEY_PATH rather than embedding it.
That is a real failure from my own run and it is worth understanding rather than deleting your way past. web-h4x2p is a generated Route. An Ingress manifest came across in the OADP restore back in Part 18, OpenShift converted it silently, and it has been quietly claiming shop.example.com ever since. Nothing broke because DNS still pointed at NSX-T, so nobody noticed. Delete the Ingress, not the Route, or the controller simply rebuilds it.
Below is the mapping I now keep open during every one of these conversions. It is the artifact worth saving out of this Part, because roughly a third of the rows have no clean equivalent and you want to discover that at design time rather than at 02:00.
| TKGI side, Ingress or NCP | OpenShift equivalent | What to watch |
|---|---|---|
| Ingress rule host plus path | Route spec.host and spec.path | One Route per host and path pair, so a three path Ingress becomes three Routes |
| TLS secret referenced in Ingress | Route spec.tls.certificate and key | Converted Ingress defaults to edge termination when no annotation is set |
| Backend TLS re-encryption | termination: reencrypt plus destinationCACertificate | Ingress has no field for it, only the route.openshift.io/termination annotation |
| TLS passthrough to the pod | termination: passthrough | From an Ingress, set path to empty and pathType to ImplementationSpecific |
| NCP ingress class annotation | Nothing, delete it | Harmless but misleading, and it hides which controller is really serving |
| NSX-T load balancer algorithm | haproxy.router.openshift.io/balance | Options are random, source, roundrobin, leastconn; default is random, and source for passthrough |
| NSX-T persistence profile | router.openshift.io/cookie_name | Passthrough Routes cannot set cookies, they hash the source IP instead |
| Source IP allow list on the virtual server | haproxy.router.openshift.io/ip_allowlist | Space separated only, and 61 entries visible in the generated config |
| Wildcard host | wildcardPolicy: Subdomain | Only Subdomain or None exist, there is no arbitrary wildcard |
| Traffic split between two app versions | alternateBackends, up to three extra | Weights 0 to 256, in cluster Services only, all zero returns 503 |
| Non HTTP virtual server, for example 5432 | Service type LoadBalancer with MetalLB | No Route can do this; default ingress controller is 80 and 443 only |
Step 4. Match the termination mode and fix the timeout
Termination mode is usually easy: if NSX-T was doing SSL offload, you want edge. If it re-encrypted to the pod, you want reencrypt and you need the backend CA. If the application terminated its own TLS, you want passthrough and you lose cookies and HTTP header rewriting along with it. Timeout is the one that ambushes people, because nothing about it appears in any Ingress manifest you are migrating.
Two rules follow from that 504. Raise the timeout per Route rather than globally on the ingress controller, so one slow reporting endpoint does not hold connections open for every application on the cluster. And keep your external load balancer idle timeout higher than your Route timeout, which Red Hat states explicitly, otherwise the load balancer tears down a connection the router still believes is live and you get intermittent resets that are miserable to trace.
If your estate is heavier on non HTTP virtual servers than on Ingress, that changes the shape of this work considerably, and it is one of the places where landing on VKS instead of OpenShift is a reasonable call; see the TKGI to VKS guide for that path. Everything else here assumes OpenShift. For how the ingress controller and OVN-Kubernetes fit together underneath all of this, Part 13 covers the plumbing and this Part will not repeat it.
Weighted Backends and a Staged Traffic Cutover
A Route can carry up to four backends in total, one primary and three alternates, each with an integer weight between 0 and 256 that defaults to 100. Share of traffic is that backend’s weight divided by the sum of all weights. Set every weight to zero and the Route returns a plain 503, which is a genuinely useful thing to know before you try it as a drain mechanism.
Now the honest part, and it is the reason this section exists. alternateBackends only references Services, and only Services in the same namespace of the same cluster. It cannot address a Service that is still running on TKGI. Every blog post that suggests weighted Routes as your migration cutover mechanism has quietly skipped that sentence. Weighted Routes are excellent for the blue green step you do after arrival, when both versions live on OpenShift. Cross platform shifting has to happen one layer up, at whatever external load balancer can hold both a TKGI pool and an OpenShift pool at the same time.
flowchart LR C[Client] --> D[DNS shop.example.com] D --> V[External LB virtual server] V -->|weight falling| A[Pool A, NSX T virtual server] V -->|weight rising| B[Pool B, OpenShift ingress routers] A --> T[wave1 web pods on TKGI] B --> R[Route web, edge TLS] R --> O[wave1 web pods on OpenShift]
Step 5. Shift pool weight in four stages
Add the OpenShift ingress router nodes as a second pool behind the existing virtual server, health checked on /healthz at port 1936 or on your application path, and start it at weight zero. Confirm the pool is green while carrying nothing. Then move in four stages with a soak between each, watching error rate and p95 latency rather than pod status. My stage boundaries were 10, 25, 50 and 90 percent, with soaks of 45, 55 and 70 minutes.
Notice what the last bar admits. Even with a 60 second TTL set two days in advance, and with the record changed cleanly, 1.9 percent of requests were still hitting the TKGI pool 41 minutes later. Those came from two internal forwarders with a minimum cache floor configured locally and from a handful of long lived Java clients that resolved once at startup and never looked again. That residual is the whole argument for keeping the old pool alive rather than switching it off the moment the record changes.
Step 6. Move the DNS record last
By the time you touch the zone, the virtual server should already be sending 90 percent or more to OpenShift and every dashboard should be boring. Changing the A record to the OpenShift ingress VIP is then a cleanup step that removes one hop, not a cutover. Leave the old virtual server configured, with the TKGI pool at low weight, for a full 72 hours. Raise the TTL back to 3600 only after that.
Verification, Rollback and Common Failures
Verification for a cutover is a distribution, not a single request. One successful curl proves that one path worked once. What you need is a couple of hundred requests, a status code histogram, and confirmation that the Route was actually admitted by the ingress controller rather than merely accepted by the API server.
| What you see | Cause | Fix |
|---|---|---|
| route claims host that is already claimed | A generated Route from a restored Ingress already holds the hostname | Delete the Ingress, not the Route, otherwise the controller recreates it |
| Admitted=False with reason HostAlreadyClaimed | Another namespace admitted the same host first | Find it across all namespaces, then decide which project legitimately owns the name |
| 503 Application is not available | Route admitted but the Service has no ready endpoints, or the port name does not match | Check oc get endpoints and confirm the Route targetPort matches a named Service port |
| Every request 503 immediately after a weight change | All backend weights were set to zero | Restore at least one non zero weight; a Route with all weights zero is defined to return 503 |
| 504 Gateway Time-out at a consistent 30 seconds | Default HAProxy route timeout is shorter than the NSX-T virtual server was | Annotate the Route with haproxy.router.openshift.io/timeout and raise the external LB idle timeout above it |
| Ingress applied, no Route ever appears | Conversion silently declined, usually a missing explicit host or an unsupported pathType | Stop using Ingress for that workload and author the Route directly |
| Sticky sessions lost after cutover | Passthrough Routes cannot set cookies and fall back to source IP hashing | Move to reencrypt if you need real session affinity, or set an explicit cookie name on an edge Route |
| Old pool still taking traffic hours later | A resolver enforcing a local minimum cache TTL, or clients that resolve once at startup | Keep the pool warm and drain by weight; identify the client from the LB source addresses |
Field Note from a Cutover Undone by a Cached TTL
On an earlier engagement, before I had the discipline described above, we did the obvious thing. Routes were built, tested, signed off. At 22:00 we changed the A record and went to watch dashboards. Twelve minutes later the payments team called: roughly a fifth of checkout attempts were failing with a stale session error. Both platforms were serving. A user who resolved to OpenShift, got a session, then hit a resolver that still answered NSX-T on the next request, landed on a pod that had never heard of them.
We rolled the record back at 22:19. Failures did not stop. They kept arriving for another 96 minutes, because reverting a DNS change is exactly as slow as making one and we had simply doubled the number of caches holding a wrong answer. Total user visible impact was 128 minutes for what the change record described as a two minute DNS update. Nobody had made a mistake in any cluster. We had used DNS as a switch, and DNS is not a switch.
What changed after that: session state moved to a shared store before any cutover involving a stateful web tier, both platforms went behind one virtual server, and the DNS record became the last thing anyone touched rather than the first. Same migration, same applications, and the next window produced 0 failed requests across three hostnames.
Shift Weight First, Move DNS Last
My recommendation for this Part is one sentence: put both platforms behind a single load balancer virtual server, move traffic by pool weight, and treat the DNS edit as cleanup you perform after the old pool has already gone quiet. Everything else here, the Route authoring, the termination modes, the timeout annotation, exists to make that shift safe. Skip the shared virtual server and you are left with DNS as your only lever, which means your rollback plan is a cache expiry you do not control.
A clean result looks like this. Every production hostname resolves to the OpenShift ingress VIP. Every Route reports Admitted=True. No Ingress objects remain in the migrated namespaces except ones you deliberately kept. Route timeouts are set per application and sit below the external load balancer idle timeout. Any non HTTP virtual server has an explicit answer, whether that is MetalLB or an in cluster client. Old NSX-T virtual servers still exist but carry zero weight, and there is a dated ticket to remove them.
On Monday, run the Ingress inventory command from Step 1 against your own production TKGI cluster and count how many rows come back with a Service type of LoadBalancer rather than an Ingress. That number is how much of your external access has no Route equivalent at all, and it is the number that decides whether this Part takes you a week or a quarter. Part 21 picks up the secrets, config and pipelines that need to follow the traffic. Full map in the TKGI to OpenShift guide.
References
- Configuring routes, OpenShift Container Platform 4.19
- Route [route.openshift.io/v1] API reference
- Kubernetes Ingress vs OpenShift Route, Red Hat blog
About The Author
DrJha