, ,

Network Policy on VKS, NSX-T DFW to Antrea (TKGI to VKS Series, Part 16)

NSX-T distributed firewall rules do not migrate to VKS. Rebuild them as tiered Antrea policy with a working default deny, verified against a real stateful app.

TKGI to VKS Series · Part 16 of 26

Your NSX-T firewall rules do not migrate. Not one of them. That distributed firewall protecting your TKGI clusters today matches on IP sets, NSGroups and tags that Antrea has never heard of, so every rule you rely on has to be rebuilt as label based policy on the new cluster before a single workload moves. That sounds like a loss until you see how much smaller the result gets.

Who this is for: platform engineers, Kubernetes operators and VMware admins who wired identity and Harbor last part on dev-tkg-01, and now need the same microsegmentation the NSX-T distributed firewall gave the old clusters before workloads land. You have Supervisor admin rights, the kubectl vSphere plugin, and a VKS cluster running Antrea as its CNI.

TL;DR · Key Takeaways

  • NSX-T distributed firewall (DFW) rules are stateful and match on IP sets and NSGroups. Antrea policy matches on pod and namespace labels. You redesign around labels, you do not port rules one for one.
  • Antrea gives you two admin scoped CRDs, ClusterNetworkPolicy (ACNP) and Antrea NetworkPolicy (ANNP), plus ordinary Kubernetes NetworkPolicy. Admin policy runs in tiers, ordered emergency, securityops, networkops, platform, application, then Kubernetes NetworkPolicy, then baseline.
  • A single default deny Kubernetes NetworkPolicy does not reproduce a DFW default deny. Put the cluster wide default deny in the baseline tier as an ACNP, and allow DNS egress first or you break every pod.
  • Headline flow: kubectl apply an allow DNS policy, then the tiered allow rules, then the baseline default deny, and prove each path with kubectl exec and curl.
  • Tested against VCF 9.1, VKS on vSphere Supervisor, Antrea v2.6.2, kubectl v1.32.

Why NSX-T DFW rules do not travel to Antrea

Last part we wired identity, RBAC and Harbor on dev-tkg-01 in the web-app namespace. A web tier, an api tier and a PostgreSQL database now run, images pull, and developers can deploy. What they cannot do yet is trust the network, because nothing is segmenting it. On the source estate that segmentation lived in the NSX-T distributed firewall, the DFW being the hypervisor level firewall NSX-T enforces on every vNIC, and it protected the TKGI pods through NSGroups populated by NSX-T tags.

Here is the core mismatch. A DFW rule reads something like allow source NSGroup web to destination NSGroup db on TCP 5432, where each NSGroup is a dynamic membership of IP addresses that NSX-T keeps current as pods churn. Antrea does not know your NSGroups and does not care about pod IPs, which are ephemeral anyway. Antrea (the CNI that VKS installs by default, built on Open vSwitch) evaluates policy against Kubernetes labels, so the same intent becomes allow from podSelector app equals api to podSelector app equals postgres on TCP 5432. The intent survives. Mechanism, object model and every identifier are new.

That is why this is a translation, not an export. You cannot dump the DFW ruleset and load it into VKS. You read the intent out of each DFW section and express it again in Antrea, and along the way most estates find their real ruleset is far smaller than the DFW made it look, because IP based rules fan out into many entries that one label selector now covers.

Preflight before translating microsegmentation

Four things must be true before you write a single policy. Antrea is the CNI on the target cluster and its controller is healthy. Your workload pods carry consistent app labels, because label hygiene is the whole game now. You can reach the cluster as an operator. And you have exported the DFW sections that apply to the source clusters so you have the intent in front of you. The checks below prove the first three, and the export you do from NSX-T Manager or with the policy API.

# Tested against VCF 9.1, VKS on vSphere Supervisor, Antrea v2.6.2, kubectl v1.32

# confirm Antrea is the CNI and the controller is running
kubectl get pods -n kube-system -l app=antrea

# confirm the static Antrea tiers exist and their order
kubectl get tiers –sort-by=.spec.priority

# confirm your workload pods actually carry app labels
kubectl get pods -n web-app –show-labels

What green looks like: the Antrea controller and agents are Running, the five static tiers report their priorities in order, and every pod shows an app label you can select on. If a pod has no app label, stop and fix the deployment first, because a policy that selects nothing fails open and you will not notice until traffic you meant to block sails through.

$ kubectl get tiers –sort-by=.spec.priority
NAME PRIORITY AGE
emergency 50 6d
securityops 100 6d
networkops 150 6d
platform 200 6d
application 250 6d

$ kubectl get pods -n web-app –show-labels
NAME READY STATUS LABELS
api-7c9f8b6d4-2xk9q 1/1 Running app=api,tier=backend
web-6b4d9c7f5-l8m2p 1/1 Running app=web,tier=frontend
postgres-0 1/1 Running app=postgres,tier=data

Map DFW constructs to Antrea policy

This mapping table is the reference artifact for the whole segmentation cutover. I keep it open next to the exported DFW sections and tick each construct off as I express it in Antrea. Build yours from the constructs your DFW actually uses, not from a generic list, because the point is to prove every source rule has a target home before you delete anything on NSX-T.

NSX-T DFW construct Antrea equivalent Notes
NSGroup by NSX-T tag podSelector on an app label Labels replace tag membership, keep them consistent
DFW section, ordered Antrea tier plus policy priority securityops for guardrails, application for app rules
Default section allow or deny baseline tier ACNP with action Drop One Kubernetes NetworkPolicy will not do this
Allow rule, source to dest on port ACNP or NetworkPolicy ingress, action Allow Match from podSelector, not IP set
Reject with response ACNP rule action Reject Reject returns a reset, Drop is silent
Rule logging flag enableLogging on the Antrea rule Logs land on the node running the pod

Table 1. NSX-T DFW to Antrea mapping. That default section row is the one people get wrong.

Tiers are what let this stay legible. Antrea ships five static admin tiers plus a baseline tier, and an admin policy names one. Rules in a higher tier win before lower tiers are even consulted, and a policy with no tier lands in application by default. Put estate wide guardrails in securityops, put per app allow rules in application, and put the catch all default deny in baseline so it runs after any developer written Kubernetes NetworkPolicy. Traffic walks the tiers in that order until a rule matches.

flowchart TD
  A[Packet to a pod] --> B[emergency tier]
  B --> C[securityops tier]
  C --> D[networkops tier]
  D --> E[platform tier]
  E --> F[application tier]
  F --> G[Kubernetes NetworkPolicy]
  G --> H[baseline tier default deny]
  H --> I[Allowed only if a rule said Allow]
Figure 1. Antrea evaluation order. Admin tiers run first, developer Kubernetes NetworkPolicy in the middle, and the baseline default deny last so it cannot override a specific allow above it.

Convert the web tier ingress with Antrea policy

Start with the allow rules, because if you lead with a default deny you strand the app while you write them. That DFW section for the source app carried three intents worth keeping: external traffic reaches the web tier on 443, the web tier reaches the api tier on 8080, and only the api tier reaches PostgreSQL on 5432. Express the web to api allow as an Antrea ClusterNetworkPolicy in the securityops tier so it sits above app level noise.

apiVersion: crd.antrea.io/v1beta1
kind: ClusterNetworkPolicy
metadata:
name: web-to-api-allow
spec:
priority: 10
tier: securityops
appliedTo:
– podSelector:
matchLabels:
app: api
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: web-app
ingress:
– action: Allow
from:
– podSelector:
matchLabels:
app: web
ports:
– protocol: TCP
port: 8080
name: AllowWebToApi
enableLogging: true

Apply it, then confirm Antrea realized it. A realized policy shows a current node count that matches where the selected pods run, which is how you know the agents programmed the rule and it is not just sitting in the API server.

$ kubectl apply -f web-to-api-allow.yaml
clusternetworkpolicy.crd.antrea.io/web-to-api-allow created

$ kubectl get acnp
NAME TIER PRIORITY DESIRED-NODES CURRENT-NODES AGE
web-to-api-allow securityops 10 2 2 15s

Prove the api tier reaches the web tier before you move on. Two curls from inside the cluster, one that should pass and one that should still be open because you have not denied anything yet, tell you the allow landed without a surprise.

$ kubectl exec -n web-app deploy/web — curl -s -o /dev/null -w "%{http_code}n" api:8080/healthz
200

Isolate the database tier at the baseline

Now the two halves that reproduce a DFW default deny. First a tight allow that lets only the api tier reach PostgreSQL on 5432 and drops the rest, in securityops so it outranks anything an app team writes. Then a namespace wide default deny in the baseline tier, which is the piece a single Kubernetes NetworkPolicy cannot give you.

apiVersion: crd.antrea.io/v1beta1
kind: ClusterNetworkPolicy
metadata:
name: db-tier-isolation
spec:
priority: 5
tier: securityops
appliedTo:
– podSelector:
matchLabels:
app: postgres
namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: web-app
ingress:
– action: Allow
from:
– podSelector:
matchLabels:
app: api
ports:
– protocol: TCP
port: 5432
name: AllowApiToPostgres
– action: Drop
name: DropAllOtherToPostgres
enableLogging: true

Add a default deny without breaking DNS

Here is the failure that cost me the better part of a morning, and every tutorial walks you straight into it. I applied a baseline ACNP that dropped all ingress and all egress for the namespace, exactly the DFW default deny I wanted. Within seconds the app fell over. Pods could not resolve anything, because a default egress deny also blocks DNS to CoreDNS on port 53, and nothing works once name resolution dies.

$ kubectl logs -n web-app deploy/api
dial tcp: lookup postgres.web-app.svc.cluster.local on 10.96.0.10:53:
read udp 10.244.1.7:41653->10.96.0.10:53: i/o timeout

The fix is to allow DNS egress in a tier above baseline before the default deny takes hold. On the DFW you never thought about DNS because the infrastructure section allowed it globally. On Antrea you make it explicit, once, in the platform tier.

apiVersion: crd.antrea.io/v1beta1
kind: ClusterNetworkPolicy
metadata:
name: allow-dns-egress
spec:
priority: 1
tier: platform
appliedTo:
– namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: web-app
egress:
– action: Allow
to:
– namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
– protocol: UDP
port: 53
– protocol: TCP
port: 53
name: AllowDNS

apiVersion: crd.antrea.io/v1beta1
kind: ClusterNetworkPolicy
metadata:
name: web-app-default-deny
spec:
priority: 5
tier: baseline
appliedTo:
– namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: web-app
ingress:
– action: Drop
egress:
– action: Drop
Contrarian call: the common advice, one default deny Kubernetes NetworkPolicy per namespace, does not equal a DFW default deny. Kubernetes NetworkPolicy only isolates the pods it selects and is purely additive, so anything it does not select stays wide open. A baseline tier ACNP over the whole namespace is what actually closes the estate, and it must sit below your allow rules, not above them.

Verify enforcement and roll back

Enforcement is proven by what fails, not by what works. Run three probes from inside the cluster: the api reaches PostgreSQL on 5432 and passes, the web tier reaches PostgreSQL and is blocked, and every pod still resolves DNS. A blocked path with a baseline Drop times out silently, so give curl a short timeout or you sit there waiting.

# allowed path, api to postgres
$ kubectl exec -n web-app deploy/api — curl -s -m 3 -o /dev/null -w "%{http_code}n" postgres:5432
000 # 000 with exit 52 is normal, postgres speaks its own protocol, the TCP connect succeeded

# blocked path, web to postgres, should time out
$ kubectl exec -n web-app deploy/web — curl -s -m 3 postgres:5432
command terminated with exit code 28 # 28 is the curl timeout, the Drop is working

# DNS still resolves everywhere
$ kubectl exec -n web-app deploy/web — nslookup postgres.web-app.svc.cluster.local
Address: 10.96.44.12

To see what Antrea actually programmed on a node, ask the agent. antctl, the Antrea command line tool that ships in the agent pod, lists the realized policies and the pods each one is applied to, which is the fastest way to confirm a rule is live rather than merely accepted.

$ kubectl exec -n kube-system antrea-agent-x9k2p -c antrea-agent — antctl get networkpolicy
NAME APPLIED-TO RULES SOURCE-TYPE SOURCE-NAME
db-tier-isolation c2f1… 2 AntreaClusterNetworkPolicy db-tier-isolation
web-app-default-deny 9a3b… 2 AntreaClusterNetworkPolicy web-app-default-deny

Rollback stays first class, because the source TKGI clusters and their DFW are still enforcing for every workload that has not moved. Backing out on the target is a delete, and because Antrea removes the flows immediately the namespace returns to open in under a second. You are never stuck with a half applied policy set.

# back out the segmentation, remove the deny first if the app is live
kubectl delete acnp web-app-default-deny
kubectl delete acnp db-tier-isolation web-to-api-allow allow-dns-egress

# the source NSX-T DFW keeps protecting the un-migrated clusters, untouched

Error you see Likely cause Fix
no matches for kind ClusterNetworkPolicy in version crd.antrea.io/v1alpha1 Old apiVersion copied from a stale example Use crd.antrea.io/v1beta1 on Antrea v2
admission webhook denied, tier security does not exist Tier name guessed, not one of the static tiers Use securityops, networkops, platform or baseline
DNS i/o timeout after applying default deny Egress Drop also blocks CoreDNS on 53 Allow DNS egress in a tier above baseline first
Allow rule selects nothing, traffic still dropped namespaceSelector label absent on the namespace Match kubernetes.io/metadata.name, present on every namespace
Policy accepted but CURRENT-NODES is 0 podSelector matches no running pod Fix the label, confirm with kubectl get acnp

Table 2. Antrea policy failures and their remediation. Four of the five are a name that does not match reality, one apiVersion, one tier, one namespace label, one pod label.

Rules to maintain for the same app intentSource NSX-T DFW entries versus Antrea policies, reference web-app namespacerules010203040NSX-T DFW34Antrea on VKS4
Figure 2. Same intent, fewer objects. The source DFW carried 34 entries across three sections, most of them IP set permutations; the label based Antrea rewrite needed 4 policies. Fewer rules is the quiet win of moving off IP based matching.

Field note and verdict

What I got wrong first: on the pilot namespace I applied the baseline default deny and the allow rules in one kubectl apply, trusting Antrea to sort the order out. It did, but my allow rules pointed at a namespaceSelector with a label the namespace did not carry, env equals prod, which I had assumed was there. The selector matched nothing, so the allow rules were inert and only the baseline Drop had any effect. That app went dark for about 40 minutes while I chased a firewall bug that was really a missing label. I switched every namespaceSelector to kubernetes.io/metadata.name, which Kubernetes stamps on every namespace automatically, and it came straight back.
My verdict: do not try to port the DFW ruleset, redesign it around labels and you will end with a fraction of the objects. Put guardrails and tier isolations in securityops, per app allows in application, and the catch all deny in baseline, and always allow DNS egress before the deny lands. Avoid two moves, matching on IP blocks when a podSelector will do, and leaning on a single Kubernetes NetworkPolicy to fake a default deny it cannot deliver.

Translate segmentation before the first workload wave

A clean result looks like this: the Antrea tiers report in order, an allow DNS policy sits in platform, per app allows sit in securityops or application, a baseline ACNP denies the rest of the namespace, curl proves the api reaches PostgreSQL and the web tier does not, DNS resolves everywhere, and antctl shows every policy realized on the right nodes. Reach that on dev-tkg-01 and you repeat the pattern for staging and prod as each moves.

On your own estate on Monday, do one thing before you write any YAML: export the DFW sections that cover your source clusters and, using Table 1, write next to each rule the podSelector and namespaceSelector that will carry its intent. That mapping on paper, made before you touch the cluster, is what stops the missing label trap and proves nothing is orphaned when you finally decommission the DFW. For the deeper NSX and Antrea integration behind these same namespaces, the NSX complete guide and the field notes on the Antrea to NSX adapter on VKS go further than a migration runbook should. The full migration map lives on the TKGI to VKS guide. Next part starts the workload waves proper, with Velero on both estates as the migration toolchain.

Questions worth answering

Can I keep enforcing with NSX-T on VKS instead of Antrea?
You can, through the Antrea to NSX adapter that registers VKS inventory into NSX and lets you author policy centrally. That is a valid target if a security team owns firewalling estate wide. For a clean migration most teams start with native Antrea policy and add the adapter later.

Do I need ACNP, or is Kubernetes NetworkPolicy enough?
Kubernetes NetworkPolicy covers per namespace app rules that developers own. Use Antrea ClusterNetworkPolicy for anything estate wide, for tiering, and for a true default deny, none of which plain NetworkPolicy can express.

Where do dropped packets get logged?
Set enableLogging on the rule and Antrea writes to the node running the applied pod, under the Antrea log directory. There is no central pane like the DFW log unless you add the NSX adapter or ship the node logs yourself.

Should I mirror DFW rule priorities exactly?
No. Reuse the intent, not the numbers. Antrea tier order plus a small priority range per tier is easier to reason about than a flat DFW section with hundreds of sequence numbers.

This series covers a production migration. Apply any default deny in a change window against your own environment, prove DNS and the allow paths first, and leave the source DFW enforcing until the target is verified.

TKGI to VKS Series · Part 16 of 26
« Previous: Part 15  |  Guide  |  Next: Part 17 »

References

About The Author


Discover more from Journal of Intelligent Infrastructure

Subscribe to get the latest posts sent to your email.

Leave a Reply

Your email address will not be published. Required fields are marked *

Architect’s Toolkit

About the Author

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

Discover more from Journal of Intelligent Infrastructure

Subscribe now to keep reading and get access to the full archive.

Continue reading