Nearly every networking conversation I have had about this migration starts in the wrong place. Somebody opens with distributed firewall rules, because that is the part the security team owns and the part that feels hardest. It is not the hardest part. Distributed firewall policy that NCP generated from Kubernetes NetworkPolicy objects comes across for free, because those objects are Kubernetes API resources and OVN-Kubernetes enforces the identical spec with its own access control lists. You reapply the YAML and you are done.

What does not come across is everything NCP quietly did that was never expressed as a Kubernetes object at all. Each namespace got a dedicated NSX-T segment carved from a pod IP block. Each namespace got a Tier-1 router. Each namespace got its own source NAT address from an external pool, which meant that for four years your firewall team, your partner integrations and your database access control lists have all been keying off a per namespace, routable, stable source IP. None of that is in a manifest. None of it is in your Git repository. It exists as NSX-T configuration that NCP created on your behalf, and on OpenShift there is no equivalent object waiting to receive it.

That asymmetry is what this assessment is for. Below is the packet path on both sides, drawn honestly, because once you see where the source address is rewritten on each platform the rest of the design decisions follow from it.

flowchart LR
  subgraph TKGI [TKGI with NCP on NSX-T]
    A[Pod on namespace segment] --> B[Namespace Tier 1 router]
    B --> C[Tier 0 router]
    C --> D[SNAT to namespace external IP]
    D --> E[Physical fabric sees namespace identity]
  end
  subgraph OCP [OpenShift with OVN Kubernetes]
    F[Pod on node host subnet] --> G[Integration bridge br int]
    G --> H[Geneve tunnel between nodes]
    H --> I{Namespace matched by an EgressIP}
    I -- No --> J[SNAT to node IP]
    I -- Yes --> K[SNAT to egress IP on assignable node]
    J --> L[Physical fabric sees a node]
    K --> M[Physical fabric sees the egress IP]
  end

Address Space, SNAT and Egress Identity

Start by measuring what NCP actually consumed, because the numbers are usually larger than anyone remembers. On the reference estate, three clusters across dev, staging and production, NCP had carved a slash 24 out of the pod IP block for every namespace and burned one external address per namespace on top of that. Nobody had ever counted them in one place.

Sixty one namespaces, sixty one source NAT rules, sixty one external addresses. That last figure is the one to carry into the OpenShift design, and it is also the one that will mislead you if you take it at face value. I will come back to that.

OVN-Kubernetes divides address space along a completely different axis. You declare one cluster network CIDR at install time and a host prefix, and the plugin carves a subnet per node rather than per namespace. A slash 14 cluster network with a slash 23 host prefix gives you room for a large number of nodes and roughly 510 pods each, and namespaces are simply not a unit of address allocation at all. Pods have no presence on the physical fabric. Anything outside the cluster that used to reach a pod IP directly, a monitoring probe, a legacy agent, a partner health check, has to move to a Service name or a Route.

Where you do need a stable outbound identity, OpenShift gives you the EgressIP object. You label one or more nodes as eligible to host egress addresses, then create an EgressIP that binds a set of addresses to a namespace selector. OVN-Kubernetes attaches the address to the selected node and rewrites the source address for matching pods. Getting it wrong the first time is almost a rite of passage, and the failure is silent rather than loud.

Two things about that sequence bite people. First, an EgressIP with no assignable node is accepted by the API server and then does nothing at all, so your pods keep egressing as the node and your firewall keeps rejecting them while every object looks healthy. Second, the address must live in the subnet of the node primary interface unless you deliberately configure a secondary interface for it, which means your network team needs to reserve egress addresses out of the node subnet rather than out of a separate pool the way they did for NSX-T. That conversation takes longer than the configuration.

Here is what the reference estate looked like once every networking artifact was counted and classified. This table is the input to the wave planning from Part 6, and it is worth building for your own estate before anybody draws a target architecture.

NSX-T and NCP Capability Mapping to OVN-Kubernetes

This is the artifact to keep. Print it, argue with your network team over it, and use the Travels column to decide what goes in the design document and what goes in the risk register. A Partly in that column means the capability exists on OpenShift but the configuration does not carry over, so somebody writes it again.

Ingress, Routes and Load Balancer Services

On TKGI, a Service of type LoadBalancer was almost free. NCP saw the object, asked NSX-T for a layer 4 virtual server, and an address appeared. Developers used it for anything they wanted reachable, including plain HTTP services that had no business consuming a virtual server at all. Broadcom documentation is explicit that load balancer size is fixed at cluster creation and cannot be changed afterwards, and that TKGI needs a large edge node virtual machine or a bare metal edge because of how many virtual servers it demands. Most estates I have looked at have quietly outgrown their original sizing and nobody wants to say so.

OpenShift inverts that economy. An installer provisioned cluster on vSphere brings up two virtual addresses maintained by keepalived, one for the API and one for the applications wildcard, and behind the applications address sits the default IngressController running HAProxy. HTTP and HTTPS workloads reach the outside world through Route objects on that shared ingress, which costs you nothing per service. There is no vSphere cloud provider that fulfils a Service of type LoadBalancer, so if you genuinely need a layer 4 address for a non HTTP protocol you install the MetalLB Operator and give it an address pool.

That reframes the 34 LoadBalancer Services in the inventory. Classify each one by protocol rather than by object type. On the reference estate, 30 of them were HTTP or HTTPS and became Routes, which removed 30 NSX-T virtual servers from the estate and saved the edge cluster more capacity than any tuning exercise ever had. Four were real layer 4, a PostgreSQL replica endpoint, two message brokers and a syslog collector, and those got a MetalLB address pool. Nobody missed the other 30, and two teams thanked me because a Route gave them a hostname instead of an address they had been hard coding into configuration files.

Ingress objects need one deliberate decision each. OpenShift accepts a standard Kubernetes Ingress and converts it into a Route automatically, so the migration appears to work with no edits. What the conversion cannot infer is your TLS intent. A Route terminates TLS at the edge, passes it through untouched, or re-encrypts to the backend, and those three behaviours have very different consequences for a service that was doing its own certificate handling behind an NSX-T virtual server. Decide the mode per service during assessment, in a spreadsheet, before anybody runs a single command. Discovering it during cutover is how a 20 minute change window becomes an evening.

MTU and Encapsulation Overhead

OVN-Kubernetes tunnels traffic between nodes with Geneve, and Red Hat documents the overhead as 100 bytes. Cluster MTU therefore has to be 100 less than the lowest hardware MTU anywhere in the cluster. On a plain 1500 byte fabric that means pods get 1400. On TKGI with jumbo frames configured on the NSX-T tunnel endpoints, pods saw a full 1500 and nobody ever thought about it, which is precisely why this catches teams who assume parity.

Health checks passed, readiness probes passed, dashboards stayed green, and a batch endpoint silently timed out for anything over roughly 1400 bytes of payload. Encapsulated frames were exceeding the physical MTU and getting dropped between nodes, and because path MTU discovery was blackholed by an intermediate device nothing reported it. Two engineers spent most of a day on that before somebody thought to compare payload sizes.

Field Note from an Egress Audit Across Three Clusters

I designed this wrong first, and it cost real time. Sixty one namespaces had sixty one dedicated source addresses on NSX-T, so my first target design had sixty one EgressIP objects and a request to the network team for sixty one addresses out of the worker node subnet. Six days of work went into that document, including a naming convention and a spreadsheet nobody will ever read again.

Then the network lead asked a question I could not answer. Which of those sixty one addresses appears in an actual rule on an actual firewall. I did not know, because NCP had allocated them automatically whether anything downstream cared or not. Getting the answer took a day and a half of exporting rule sets from two perimeter firewalls and a database access control list, then grepping them against the SNAT rule table pulled from NSX-T.

Nine namespaces were referenced. Those nine clustered into four upstream dependencies, a payments partner, a mainframe gateway, an external identity provider and a regulatory reporting feed, so four EgressIP objects with label based namespace selectors covered every one of them. My sixty one address request became four. The other fifty two namespaces now egress as the node address and nothing downstream noticed, because nothing downstream had ever been looking.

One namespace did bite, and it was not in the nine. A reporting service had a partner allowlist hard coded in a configuration map rather than enforced at any firewall, so it never appeared in a rule export. Forty minutes into the pilot cutover it started failing calls to an external pricing feed, because the partner was allowlisting our old SNAT address on their side and we had no visibility into their configuration at all. Adding that namespace to an existing egress group took ten minutes; getting the partner to update their allowlist took nine days. Ask every application owner one blunt question during assessment. Does anybody outside this company have your source address written down anywhere. Firewall exports will not tell you that.

Keep OVN-Kubernetes, Audit Egress Before You Design

There is a tempting alternative I want to name and then reject, because somebody on your team will raise it. NCP has an OpenShift build. You could run it as the network plugin on OpenShift, keep NSX-T in the data path, keep your distributed firewall rules, keep pod IPs routable, and keep the network team on tooling they already know. On paper it looks like the cheapest possible networking migration.

Do not do it. Choosing NCP puts every OpenShift minor upgrade behind a third party compatibility matrix, which is exactly the coupling you are leaving TKGI to escape. Part 1 established that NSX 9.1 removed the legacy management plane API and that NCP dropped TKGI support as a result; the same forcing function applies to any NCP integration you build on. Beyond version risk, OVN-Kubernetes is where Red Hat ships and tests everything that matters going forward, including EgressIP, EgressFirewall, AdminNetworkPolicy and network observability. A supported path you have to learn beats a familiar path that constrains your upgrade cadence, and the learning is smaller than it looks because the Kubernetes objects are identical on both sides.

My recommendation for this phase is narrow and specific. Take OVN-Kubernetes as given, because it is the default and the argument against it is weaker than it feels. Fix cluster MTU before install, from an end to end measurement rather than from what TKGI displayed. Reclassify all 34 of your LoadBalancer Services by protocol and expect most of them to become Routes. Then spend your real design effort on egress identity, because that is where the only genuinely irreplaceable NCP behaviour lives.

On Monday, do one thing on your own estate. Export the SNAT rule table from NSX-T, export the rule sets from every perimeter firewall and database access control list you can reach, and cross reference them. Whatever number comes back is your real EgressIP design, and it will be far smaller than your namespace count. Part 9 turns to storage, where vSphere CSI, persistent volumes and reclaim policy decide how much of your data actually moves.

References

• Configuring an egress IP address, OVN-Kubernetes network plugin, OpenShift Container Platform, Red Hat Documentation
• Changing the MTU for the cluster network, Advanced networking, OpenShift Container Platform, Red Hat Documentation
• Size a Load Balancer, network profiles, Tanzu Kubernetes Grid Integrated Edition, Broadcom TechDocs
• Load balancing with MetalLB, Networking, OpenShift Container Platform, Red Hat Documentation

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts