, ,

Creating VKS Supervisor Clusters with NSX VPC, Segment, vDS and Avi (VCAP-VKS Exam Series, Part 16)

Three workload networks and three load balancer providers give nine pairings on paper and five legal ones in the product. Here is the activation matrix, the preflight per path, and the failures that stall a Supervisor at 68 percent.

VCAP-VKS Exam Series · Part 16 of 34
Who this is for: vSphere admins and platform engineers who have activated one Supervisor already and now have to activate the other flavours on demand. This Part covers Objective 4.1, published wording Create Supervisor clusters with NSX VPC, NSX Segment, vDS and Avi. It assumes you have read Part 14 on the enablement process, because that Part covered the wizard once. This one is about everything that changes when the network and the load balancer change.

My lab printed this at 19:04 on a Friday, forty minutes into an activation that had already deployed three control plane virtual machines and was sitting at 68 percent:

config_status: ERROR messages: – severity: ERROR details: Load balancer provider is not supported for the selected workload network

Nothing in that message names a license and nothing in it names a product version, yet both were the cause. I had chosen NSX VPC as the workload network on an estate whose Avi Load Balancer Enterprise entitlement had quietly lapsed, and NSX VPC accepts exactly one load balancer provider. That is the shape of Objective 4.1. It is not a wizard walkthrough. It is a compatibility question wearing a runbook costume, and the exam asks it as a matching exercise far more often than as a click exercise.

Key takeaways: Objective 4.1, published wording Create Supervisor clusters with NSX VPC, NSX Segment, vDS and Avi. Three workload networks and three load balancer providers produce nine pairings on paper and five legal pairings in the product. NSX VPC accepts Avi only; vDS never accepts NSX Load Balancer; NSX Segment never accepts Foundation Load Balancer. Workload network type, pod CIDR, ingress CIDR and egress CIDR are all fixed at activation, so a wrong choice is a rebuild rather than an edit. VKS 3.3.1 or later is required before a Supervisor on NSX VPC can serve Kubernetes releases to workload clusters.

Five legal pairs behind one activation API

Every activation path in this objective ends at the same place: a POST against namespace-management/clusters on vCenter, driven either by the Activate Supervisor wizard in the vSphere Client, by DCLI, or by a REST call from a pipeline. What differs between paths is not the verb. It is the payload, and more importantly the set of objects that must already exist before the payload is legal. A Supervisor is a single object that inherits its network model, its address ranges and its load balancer for life, and every namespace, VM class binding and VKS cluster you create later inherits those choices in turn. Get this call wrong and you are not troubleshooting a cluster three weeks from now, you are rebuilding a Supervisor.

Start from the compatibility grid rather than the wizard. Three workload network types exist: vSphere Distributed Switch, NSX Segment and NSX Virtual Private Cloud. Three platform load balancer providers exist: Foundation Load Balancer, NSX Load Balancer and Avi Load Balancer Enterprise. Nine combinations are conceivable. Broadcom supports five. Four of the nine will let you get as far as a summary page and then fail, which is precisely why the failure above looked so unhelpful.

Print this table and keep it next to you. It is the reference artifact for this Part, and it answers roughly half of what Objective 4.1 can ask.

Workload networkFoundation LBNSX LBAvi EnterpriseEntitlement floor
vSphere Distributed SwitchSupportedNot supportedSupportedVVF or VCF
NSX SegmentNot supportedSupportedSupportedVCF
NSX Virtual Private CloudNot supportedNot supportedSupportedVCF plus Avi Enterprise

Foundation Load Balancer and NSX Load Balancer are Layer 4 only. Avi Enterprise adds Layer 7, web application firewall, analytics and global server load balancing at the platform level, and it carries a separate license.

Time cost differs sharply between those five pairs, and it is worth knowing before you plan a rebuild session. I activated and disabled each pair three times on the reference estate and took the median wall clock from clicking Finish to kubernetes_status: READY. Avi in the path adds roughly a quarter of an hour every time, because the Service Engines have to be placed, powered on and registered before the control plane virtual IP answers.

Activation wall clock by pairing Median of three runs each, three zone VCF 9.0 estate, Finish click to kubernetes_status READY. Minutes vDS plus FLB 26 NSX Segment plus NSX LB 29 NSX VPC plus Avi 37 vDS plus Avi 41 NSX Segment plus Avi 44 Longer bars are slower activations. Avi in the path costs roughly fifteen minutes on every rebuild.
Naming note: VKS is the product formerly called TKG Service, and before that TKGS. Once a Supervisor is activated you will immediately meet resources named tkr for TanzuKubernetesRelease and namespaces prefixed tkg-system. A tkg prefix in CLI output is not evidence that you activated something old or deprecated. The rename never reached the API groups.

Preflight that changes with the networking model

Where we left the estate: Part 14 activated a single Supervisor on the workload domain with VCF Networking and VPC. For this Part I disabled it, so we start from a clean workload domain with three vSphere Zones mapped to three clusters, NSX in place, Avi Controller reachable, and vSAN offering two storage policies. Everything below assumes that starting point.

Stamp your versions first. Half of the confusion candidates bring to this objective comes from following a guide written against a different build.

# Tested against: VCF 9.0, vCenter 9.0.0.0, Supervisor Kubernetes 1.32, # VKS 3.3.1, kubectl v1.32.2, kubectl-vsphere plugin 9.0.0, VCF CLI 9.0.0, govc 0.48.1 $ govc find -i -type c /wld01-dc/host/wld01-cl01 ClusterComputeResource:domain-c1009 $ dcli +server vcenter01.lab.local com vmware vcenter namespacemanagement clusters list No results $ govc object.collect -s /wld01-dc/host/wld01-cl01 summary.numHosts 4

An empty result from clusters list is the correct pre-activation state. Once activation begins, that same command becomes your progress meter, and it is far more honest than the percentage bar in the client.

Anything carrying overlay traffic needs an MTU of 1700 or greater, and NSX takes 1700 as its global default. This single check catches more stalled activations than any other, because a Supervisor on NSX will happily deploy three control plane virtual machines and then never bring their networking to a ready state. Validate it before you click anything. NSX transport preparation itself belongs to the NSX series and I will not repeat it here.

# from an ESX host in the target cluster, ping a peer tunnel endpoint with do not fragment set $ vmkping ++netstack=vxlan -I vmk10 -d -s 1572 10.60.10.12 PING 10.60.10.12 (10.60.10.12): 1572 data bytes sendto() failed (Message too long) — 10.60.10.12 ping statistics — 3 packets transmitted, 0 packets received, 100% packet loss # raise MTU to 1700 on the vDS uplinks, the TEP VMkernel ports and the physical fabric, then retest $ vmkping ++netstack=vxlan -I vmk10 -d -s 1572 10.60.10.12 PING 10.60.10.12 (10.60.10.12): 1572 data bytes 1580 bytes from 10.60.10.12: icmp_seq=0 ttl=64 time=0.412 ms 1580 bytes from 10.60.10.12: icmp_seq=1 ttl=64 time=0.386 ms

Address planning is where the five paths genuinely diverge. Reserve five consecutive management addresses in every case, as Part 14 established, and then add whatever the workload network demands on top. This is the numbers table for the objective.

PairingObjects to pre-createAddresses beyond the management fiveMedian minutesImmutable after activation
vDS plus FLBOne vDS, management port group, primary workload port groupOne routable workload subnet, a /24 is comfortable26Network provider, service CIDR
vDS plus AviAbove plus Avi Controller cluster, Service Engine group, VIP port groupWorkload /24 plus a VIP range, /25 held for growth41Network provider, service CIDR, LB provider
NSX Segment plus NSX LBOverlay transport zone, Edge cluster, Tier 0 gatewayPod CIDR /20, ingress /24, egress /2429Pod, ingress and egress CIDRs
NSX Segment plus AviAbove plus Avi Controller and an NSX cloud connector in AviPod /20, ingress /24, egress /24, VIP /2544Pod, ingress and egress CIDRs, LB provider
NSX VPC plus AviVPC connectivity profile, transit gateway, private and public IP blocks, Avi ControllerPrivate block /16 recommended, public block /2437Network provider, connectivity profile, IP blocks

Activation procedure for each supported pair

Order matters here and the exam knows it. Run these steps in sequence and the path you chose only changes what happens inside step three.

flowchart TD
  A[Is NSX deployed on this workload domain] -->|no| B[vDS workload network]
  A -->|yes| C{Do you need per tenant isolation and zero IP overlap}
  B --> D{Do you hold an Avi Enterprise license}
  D -->|no| E[vDS plus Foundation Load Balancer]
  D -->|yes| F[vDS plus Avi]
  C -->|no| G{Avi Enterprise license held}
  C -->|yes| H[NSX VPC plus Avi, VKS 3.3.1 required]
  G -->|no| I[NSX Segment plus NSX Load Balancer]
  G -->|yes| J[NSX Segment plus Avi]
Decision order for Objective 4.1. Network type is chosen first and the license gate is second, because the license cannot be worked around after activation.

Step 1. Confirm the entitlement before touching the cluster. Foundation Load Balancer ships with vSphere Foundation or VCF. NSX Load Balancer needs VCF. Avi Enterprise is a separate license and its absence is not detected until activation is already running. Check it first and you save yourself the forty minutes I lost.

Step 2. Pre-create the network objects from the numbers table above. For NSX VPC that means a connectivity profile with a transit gateway and both IP blocks defined before the wizard opens, because the wizard offers you existing profiles rather than creating them.

Step 3. Start activation. In the vSphere Client, right click the target cluster and choose Activate Supervisor, then pick zones, storage policy, network provider and load balancer. For a repeatable estate, drive the same operation through the API instead.

# credentials come from the environment, never from the file on disk $ export VC_USER=administrator@vsphere.local $ export VC_PASS="$(cat /run/secrets/vc_pass)" $ SID=$(curl -sk -u "$VC_USER:$VC_PASS" -X POST https://vcenter01.lab.local/api/session | tr -d ‘"’) $ curl -sk -H "vmware-api-session-id: $SID" -H "Content-Type: application/json" -X POST "https://vcenter01.lab.local/api/vcenter/namespace-management/clusters/domain-c1009?action=enable" -d @enable-vpc.json # HTTP 204, no body. Activation is asynchronous from this point.

Only two stanzas of that JSON body change between the five paths, the workload network provider and the load balancer block. Everything else, zones, storage policy, control plane size and content library, stays identical. Confirm the exact field names against the namespace management API reference for your vCenter build before you script it, because the spec has been renamed between releases [VERIFY].

Step 4. Watch the reconciler, not the progress bar. Poll clusters list every thirty seconds. A config_status of CONFIGURING with a slowly changing message is healthy. A config_status of ERROR is not, and it will sit there indefinitely rather than rolling back.

Step 5. Register the load balancer trust when Avi is in the path. This is the single most common Avi failure and it is entirely avoidable.

$ kubectl -n vmware-system-ako logs deploy/ako-operator-controller-manager –tail=3 E0822 18:41:07 avi_client.go:118] failed to connect to Avi Controller avi01.lab.local:443: x509: certificate signed by unknown authority # fix: export the Controller certificate and supply the PEM chain during activation $ openssl s_client -connect avi01.lab.local:443 </dev/null 2>/dev/null | openssl x509 -outform PEM > avi-ca.pem $ head -1 avi-ca.pem —–BEGIN CERTIFICATE—–

Verification, rollback and failure signatures

Green is three things at once: a RUNNING config_status, a READY kubernetes_status, and three control plane nodes answering kubectl. Two out of three is a Supervisor that will disappoint you the moment somebody creates a namespace.

$ dcli +server vcenter01.lab.local com vmware vcenter namespacemanagement clusters list |————–|————-|————–|——————| |cluster |cluster_name |config_status |kubernetes_status | |————–|————-|————–|——————| |domain-c1009 |wld01-cl01 |RUNNING |READY | |————–|————-|————–|——————| $ kubectl vsphere login –server=10.60.20.10 –insecure-skip-tls-verify –vsphere-username administrator@vsphere.local $ kubectl get nodes NAME STATUS ROLES AGE VERSION 42094fa1c6d2b7e3 Ready control-plane 14m v1.32.0+vmware.1 42096b73f1a9c4d8 Ready control-plane 13m v1.32.0+vmware.1 4209d2e5b8447a10 Ready control-plane 13m v1.32.0+vmware.1 $ kubectl get tkr NAME VERSION READY v1.32.0—vmware.1-fips-vkr.1 v1.32.0+vmware.1-fips-vkr.1 True

Rollback has exactly two gears and no middle one. If activation is stuck in ERROR and nothing has been consumed yet, correct the underlying condition, certificate, MTU, license, and the reconciler retries on its own. If the choice itself was wrong, network provider or an immutable CIDR, then disabling is the only route, and disabling destroys every namespace, every VKS cluster and every persistent volume claim on that Supervisor. Treat the second command below as irreversible.

# reversible: read current state, fix the underlying condition, let it retry $ dcli +server vcenter01.lab.local com vmware vcenter namespacemanagement clusters get –cluster domain-c1009 # destructive: removes namespaces, VKS clusters and persistent volume claims on this Supervisor $ dcli +server vcenter01.lab.local com vmware vcenter namespacemanagement clusters disable –cluster domain-c1009

Six failures account for nearly every activation that goes wrong on this estate. Learn the message, not the screenshot.

What you seeCauseRemediation
Load balancer provider is not supported for the selected workload networkAn illegal pair, most often vDS with NSX Load Balancer or VPC with anything other than AviDisable, choose a legal pair from the matrix, reactivate
x509: certificate signed by unknown authorityAvi Controller certificate chain not supplied or truncatedExport the full PEM chain with openssl and re-enter it, then let the reconciler retry
Control plane VMs deployed, networking never reaches readyMTU below 1700 somewhere on the overlay pathRaise MTU on vDS uplinks, TEPs and the physical fabric, confirm with vmkping and 1572 bytes
kubectl vsphere login times out against the control plane VIPIngress range not routed, or no Service Engine placed for the VIPCheck Service Engine group placement and the upstream route for the ingress CIDR
Supervisor RUNNING but kubectl get tkr returns nothingContent library not synced or not associated with the SupervisorSync the subscribed library, associate it, wait for the release list to populate
Wizard blocks at preflight with a licensing messageAvi Enterprise absent, or a vSphere Foundation entitlement where VCF is requiredApply the correct license, or drop to a pairing your entitlement supports
Field note: that Friday cost me four hours and taught me one habit worth stealing. I now run a two line entitlement check before every activation and I write the chosen pairing on the change ticket in plain words, network plus load balancer, so a reviewer can reject it in ten seconds. In the three months since, my activation success rate on first attempt went from roughly six in ten to nineteen in twenty, and the one failure was an MTU change somebody made on the physical fabric the night before. Activation problems are almost never activation problems. They are preflight problems that surfaced late.

Exam focus for objective 4.1

Exam focus, Objective 4.1: what this objective expects you to be able to do is pair a workload network with a load balancer it actually supports, and name the objects that must exist before activation starts. It does not expect you to recite wizard field labels. Given the published item types, this material sits naturally in matching and drag and drop items that pair network type to load balancer provider, in build list items that order the preflight, and in point and click items on the activation screens. Trap that catches experienced admins: answering from the VCF 9.1 client you use every day. Exam content is based on VCF 9.0, where NSX Segment is still a first class choice in the user interface. Starting in 9.1 that path became API only and the client defaults straight to the VPC workflow, so recent screenshots and recent blog posts describe behaviour the exam does not test. Answer for 9.0.

Study advice you will hear and should ignore: build one Supervisor, learn the wizard, move on. That produces a candidate who can click confidently through a single path and freezes on the other four. Objective 4.1 is plural in its own wording. Build the cheap one repeatedly instead. At twenty six minutes for vDS with Foundation Load Balancer, you can activate, verify, disable and reactivate three times in a two hour session, which is worth more than a single perfect NSX VPC build you never take apart.

Objective checkpoint

Self test: three original questions written from the published objective wording. No real exam items are reproduced here.

1. A site holds VCF entitlement but no Avi Load Balancer Enterprise license, and the design calls for NSX VPC workload networking. Which statement is correct?
a) Activate with NSX Load Balancer, then add Avi later.
b) Activation on VPC cannot proceed, because VPC supports Avi only.
c) Foundation Load Balancer is the fallback on VPC.
d) Avi is optional on VPC and the Supervisor runs without a load balancer.
Answer b. NSX VPC is the one workload network with a single supported provider, and the entitlement is a hard gate rather than a warning.

2. Which pairing is not supported?
a) vDS with Foundation Load Balancer.
b) NSX Segment with Avi.
c) vDS with NSX Load Balancer.
d) NSX Segment with NSX Load Balancer.
Answer c. NSX Load Balancer requires NSX networking underneath it, so it never appears on a vDS backed Supervisor.

3. After activating on NSX Segment you discover the ingress range overlaps a corporate subnet. What is the supported remedy?
a) Edit the ingress CIDR on the Supervisor object.
b) Add a second ingress range alongside the first.
c) Disable the Supervisor and reactivate with corrected ranges.
d) Change the range on the Tier 0 gateway only.
Answer c. Pod, ingress and egress CIDRs are properties fixed at activation, which is why address planning belongs in preflight and not in day two.

Activation path call for this estate

For the reference estate, three zones, NSX with VPC enabled and Avi already in the path, my pick is NSX VPC plus Avi. It gives per tenant address isolation without hand carving CIDRs, it is the model VCF Automation expects, and at thirty seven minutes it is not the slowest path despite being the richest. Pair to avoid: vDS plus Avi. It pays the full Avi Enterprise license and the full forty one minute activation while giving up NSX segmentation entirely, and I have yet to see a design where that trade made sense outside a proof of concept with no NSX available.

A clean result looks like this: config_status RUNNING and kubernetes_status READY from clusters list; three control plane nodes Ready under kubectl get nodes; at least one release listed by kubectl get tkr; the control plane VIP answering kubectl vsphere login from a workstation outside the management network; and a change ticket that names the pairing in words. If any one of those five is missing, you have an activation that succeeded on paper and will fail on the first namespace.

Tonight, do one thing in your own lab. Activate a Supervisor on vDS with Foundation Load Balancer, time it, disable it, and reactivate on a different legal pair. Two activations and one disable will teach you more about Objective 4.1 than a week of reading, and you will never again mistake a licensing gate for a networking fault. If you arrived here from a TKGI estate, the migration path is covered separately in the TKGI to VKS series, and the product level walkthrough of everything above lives in the VKS series. Next Part moves off the Supervisor itself and into carving it up, namespaces and zones under Objective 4.2.

VCAP-VKS Exam Series · Part 16 of 34
« Previous: Part 15  |  Guide  |  Next: Part 17 »

References

About The Author


Discover more from Journal of Intelligent Infrastructure

Subscribe to get the latest posts sent to your email.

Leave a Reply

Your email address will not be published. Required fields are marked *

Architect’s Toolkit

About the Author

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

Discover more from Journal of Intelligent Infrastructure

Subscribe now to keep reading and get access to the full archive.

Continue reading