, ,

Reference Architecture for a VKS Deployment, NSX, VDS and Zones (VCAP-VKS Exam Series, Part 4)

Objective 1.3 asks you to pick a reference architecture, not describe one. Here are the four published vSphere Zone models, the four Supervisor networking paths, and the single decision you can never take back.

VCAP-VKS Exam Series · Part 4 of 34

A customer enabled a Supervisor on one vSphere Zone in April because the workload domain was ready and the hardware for the other two zones was still in a crate. In July they needed cluster level high availability for the control plane, found out that the management zone count is fixed at activation, and rebuilt the platform from scratch. Objective 1.3 exists because that mistake is common enough to be worth testing.

Key takeaways:
• Objective 1.3 is published as Determine the reference architecture (NSX, VDS, Zones) for a VKS deployment. Determine, not describe. Items give you constraints and want an architecture back.
• Broadcom publishes four named vSphere Zone models plus a simplified model. Learn the names, because they are how the documentation and the item writers talk about the same picture.
• One management zone or three management zones is chosen at Supervisor activation and cannot be changed afterwards. Workload zones can be added later. That asymmetry is the whole objective.
• Four networking paths exist in VCF 9.0: VDS with the Foundation Load Balancer, VDS with Avi, NSX segments with Tier 0 and Tier 1 routers, and NSX VPC. Only NSX VPC gives tenants self service networks, and it needs VKS 3.3.1 or later.
• Reserve five IP addresses for the Supervisor control plane on the management network. Three for the VMs, one floating, one held back for patching.
Who this is for: You read a cluster back to its Supervisor objects in Part 3. Now you sit on the other side of the table and design the estate those objects live in. I assume you can build a VDS and read an NSX topology. Component mechanics belong to the NSX complete guide and the vSphere Kubernetes Service complete guide. This part is about the choices and their consequences.

One naming note before the design work starts, because it shows up in every command later in this series. VKS, vSphere Kubernetes Service, is the product formerly called TKG Service or TKGS. Broadcom changed the product name and left the plumbing alone, so zone objects still answer to an API group called topology.tanzu.vmware.com and half the documentation URLs still say tkg. Read tanzu and tkg as VKS and move on.

flowchart TD
  A[VKS design brief] --> B{Cluster level HA required for the Supervisor}
  B -->|Yes| C[Three management zones]
  B -->|No| D[Single management zone]
  C --> E{Tenants must not share hosts with the control plane}
  D --> E
  E -->|Yes| F[Isolated workload zones]
  E -->|No| G[Combined workload zones]
  F --> H{Self service tenant networks required}
  G --> H
  H -->|Yes| I[NSX VPC with VKS 3.3.1 or later]
  H -->|No| J{NSX already deployed in the workload domain}
  J -->|Yes| K[NSX segments with Tier 0 and Tier 1]
  J -->|No| L[VDS with Foundation Load Balancer or Avi]
Decision order for objective 1.3. Availability first, isolation second, networking last, because the networking stack can be rebuilt and the management zone count cannot.

Zone models Broadcom publishes, and what each one survives

A vSphere Zone is a named container that maps to exactly one vSphere cluster and is treated as an independent failure domain. That is the whole definition. Everything interesting comes from how many of them you point the Supervisor control plane at, and how many of them you then hand to vSphere Namespaces for workloads.

Broadcom splits this into two independent axes in the VCF 9.0 design library, and candidates who have not read that library tend to collapse them into one. Axis one is management zones: one or three, decided at activation. Axis two is workload zones: combined with the management zones, or isolated into separate zones so tenant workloads never land on hosts running control plane VMs. Two axes with two settings each gives the four published models, and they are named exactly as follows.

Published modelManagement zonesMinimum clustersSurvivesChangeable later
Single Management Zone with Combined Workload Zones11Host failure, via vSphere HANo for management, yes for adding workload zones
Single Management Zone with Isolated Workload Zones12Host failure, plus noisy tenants kept off control plane hostsNo for management, yes for more workload zones
Three Management Zones with Combined Workload Zones33Loss of one entire vSphere clusterNo for management, yes for more workload zones
Three Management Zones with Isolated Workload Zones34Loss of one entire vSphere cluster, with tenant isolationNo for management, yes for more workload zones

Print that table. It is the reference artifact for this part, and every design item on the exam that mentions zones is asking you to land on one of those four rows. A fifth option, the Simplified Supervisor Model, exists in the same design library for people who want a Supervisor with the fewest decisions possible, and it is a single management zone underneath. Treat it as an enablement workflow rather than a distinct architecture.

Two rules govern how workload zones attach. A vSphere Namespace requires at least one vSphere Zone, and you can assign up to three. Assign three and the Supervisor spreads that namespace across all three underlying clusters in equal parts. Broadcom gives the arithmetic plainly: dedicate 300 MHz of CPU to a namespace on a three zone Supervisor and 100 MHz is taken from each vSphere cluster. That is not a rounding detail. It means a single pod in a three zone namespace can never consume the whole reservation, and it catches people who size namespaces the way they size resource pools.

Networking stack choice and what it locks in

Zones decide where things run. Networking decides what tenants can ask for. VCF 9.0 ships four supported paths to a Supervisor, and the exam guide names NSX and VDS in the objective wording precisely because candidates who have only ever seen one of them tend to answer from habit.

PathLoad balancingTenant self service networksPick it when
VDS with Foundation Load BalancerLayer 4 onlyNoNo NSX in the workload domain and the estate is a lab, a proof of concept or an edge site
VDS with Avi Load BalancerLayer 4 and Layer 7, with AKO as ingressNoYou need Layer 7, WAF or DNS integration but the workload domain has no NSX
NSX segments with Tier 0 and Tier 1NSX edge load balancer at Layer 4, or Avi alongsideNo, the administrator carves segmentsNSX is already deployed, vSphere Pods are required, and tenants do not need to define their own subnets
NSX VPCNSX or Avi, chosen at enablementYes, with automatic routing, NAT and subnetsMultiple tenants, and the Supervisor is 9.0 or later with VKS 3.3.1 or later

Three things about this table earn their place on a score report. Foundation Load Balancer is Layer 4 only, so any requirement mentioning ingress termination, a web application firewall or certificate offload rules it out immediately. NSX VPC is the only row that gives tenants a self service network boundary in the public cloud sense, and it carries a hard version floor: Supervisor 9.0 introduced VPC support and VKS 3.3.1 or later is required to consume it. And vSphere Pods, the CRX backed pods that run directly on ESX, need NSX. If a scenario asks for vSphere Pods and offers you a VDS answer, the VDS answer is wrong regardless of how sensible the rest of it looks.

What the networking choice does not lock is as important as what it does. Unlike the management zone count, a networking stack can be replaced by rebuilding the Supervisor on the same hardware, which is expensive but not architectural surgery. I have watched teams treat both decisions as equally permanent and consequently overthink the load balancer while under thinking the zones. Get the zones right first.

Host, IP and capacity arithmetic per model

Design items on this exam are frequently arithmetic in disguise. A scenario hands you a host count and asks which model fits, and the candidate who has memorised the per zone minimums answers in ten seconds while everybody else reasons from first principles and runs out of clock.

Broadcom states the per zone floor plainly. Each vSphere cluster backing a zone needs at least three ESX hosts, and four when vSAN provides the storage. Test and proof of concept estates are allowed to drop to one host, or two with vSAN, which is why so many home lab builds look nothing like the production numbers in the design library. Multiply the vSAN floor across the four models and the capital cost of each row becomes obvious.

Minimum ESX hosts per zone model, vSAN storageVCF 9.0 production floor of four hosts per vSAN cluster, multiplied across the required clusters.1 mgmt zone, combined41 mgmt zone, isolated83 mgmt zones, combined123 mgmt zones, isolated16Proof of concept estates may drop to two hosts per vSAN cluster, which is why lab numbers mislead.

IP arithmetic is smaller and gets forgotten more often. On the management network the Supervisor control plane wants five contiguous addresses: one for each of the three control plane VMs, one floating address that follows whichever VM currently answers, and a fifth held in reserve so that a patch can stand up a replacement VM without evicting anything. Hand a Supervisor a range of four and enablement completes, then the first patch cycle stalls. Every zone can use its own management network, which is a genuine relief on stretched physical estates and a detail worth remembering because it contradicts the intuition that a Supervisor needs one flat management subnet.

Field note: Popular study advice says start with one zone and grow into three. That advice is half wrong and the wrong half is expensive. Workload zones can be added to a running Supervisor at any time, so growing tenant capacity is easy. Management zones cannot: the count is fixed when you activate the Supervisor. On the rebuild I mentioned at the top, the estate held 11 vSphere Namespaces and 4 VKS clusters, and re-creating the namespace definitions, storage policy bindings, VM class assignments and identity role bindings took me about three weeks of evenings plus a six hour cutover window. Not one line of that work was interesting. If there is any chance the platform will need to survive a cluster failure, activate on three management zones on day one, even if two of those zones start life with the bare minimum host count.

Discovery on an estate you inherited

Design work rarely starts on a blank sheet, and neither do design items. Both hand you an existing estate and ask what it is. These are the three commands I run before I write a single line of a VKS design document, against the reference lab this series uses throughout.

# Estate for every command in this part # VCF 9.0, vSphere Supervisor 9.0, three vSphere Zones on three vSphere clusters # NSX with VPC networking, Avi in the load balancing path, VKS 3.3.1 # kubectl 1.32.x with the kubectl-vsphere plugin, vSAN backing two storage policies # Credentials from the environment, never from shell history export VSPHERE_USER="admin@vsphere.local" export KUBECTL_VSPHERE_PASSWORD="$(read -rsp ‘vCenter password: ‘ p; echo "$p")" kubectl vsphere login –server="$SUPERVISOR_VIP" –vsphere-username="$VSPHERE_USER"

First question, how many zones does this Supervisor know about, and does the API agree with the vSphere Client. Zone objects live on the Supervisor as a custom resource, and the API group is the naming artefact I flagged earlier.

$ kubectl get availabilityzones NAME AGE zone-a 61d zone-b 61d zone-c 61d $ kubectl get availabilityzone zone-a -o jsonpath='{.apiVersion}{"n"}’ topology.tanzu.vmware.com/v1alpha1 # And inside a VKS cluster, where the nodes actually landed $ kubectl get nodes -L topology.kubernetes.io/zone NAME STATUS ROLES AGE VERSION ZONE vks-prod-01-control-plane-4x9qz Ready control-plane 38d v1.32.4 zone-a vks-prod-01-control-plane-hb2mn Ready control-plane 38d v1.32.4 zone-b vks-prod-01-control-plane-t7wkl Ready control-plane 38d v1.32.4 zone-c vks-prod-01-np-a-6c9f7d8b4-2pkxv Ready <none> 38d v1.32.4 zone-a vks-prod-01-np-a-6c9f7d8b4-9djqw Ready <none> 38d v1.32.4 zone-b vks-prod-01-np-a-6c9f7d8b4-mv6rt Ready <none> 38d v1.32.4 zone-c

Now the failure this design produces, which I have caused twice and which almost nobody predicts on paper. Spreading nodes across three zones is good for stateless workloads and actively hostile to a naive stateful one. Block persistent volumes are per zone. They do not replicate across zones at the storage layer, because the latency penalty would be unacceptable. So a pod that gets rescheduled into a different zone from its volume cannot start, and the scheduler tells you exactly that.

$ kubectl describe pod postgres-0 -n data … Events: Type Reason Age From Message —- —— —- —- ——- Warning FailedScheduling 47s default-scheduler 0/6 nodes are available: 6 node(s) had volume node affinity conflict. # Confirm the mismatch. The volume is pinned to one zone. $ kubectl get pv -o custom-columns=NAME:.metadata.name,ZONE:.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values NAME ZONE pvc-3f0a91c2-7b44-4d0e-9a1e-5c8d2f6b0e77 [zone-a]

Two fixes, and picking between them is a design decision rather than a repair. Constrain the StatefulSet to a single zone with node affinity and accept that a zone failure takes that database offline until you restore it, or run replication at the application layer with one replica per zone and let each replica own a volume in its own zone. High availability across zones for stateful services is an application property, not a storage property. That sentence has saved more of my designs than any diagram.

Exam focus for objective 1.3

EXAM FOCUS, objective 1.3: This objective expects you to read a set of constraints and return an architecture, then defend it. Expect matching and drag and drop item types where a requirement such as tenant self service networking, Layer 7 ingress, vSphere Pod support or survival of a full cluster outage has to be paired with the zone model or networking path that delivers it. Expect build list items that put the enablement decisions in order. Expect point and click on an architecture diagram.

The trap that catches experienced administrators is treating three zones as a straightforward upgrade from one. It is not an upgrade at all. A candidate who has run vSphere for a decade reasons that capacity can always be added later, which is true for workload zones and false for management zones, and picks a single zone answer in a scenario that says the platform must tolerate the loss of a vSphere cluster. Read every scenario for the words that describe a cluster level failure. If they are present, only a three management zone model is correct, and the correct time to choose it is before activation.

A second, quieter trap lives in the load balancer column. Requirements that mention a web application firewall, TLS termination at the edge or DNS record automation are Avi requirements, not NSX edge requirements, because the NSX load balancer that ships by default with a VCF configuration handles Layer 4 only. Candidates who work in NSX heavy shops answer NSX by reflex and lose the item.

Objective checkpoint

Three questions, written from the published objective:

1. A workload domain has four vSphere clusters of four hosts each, all on vSAN. Business requires that the Supervisor control plane survives the loss of one entire cluster, and that tenant workloads never run on hosts carrying control plane VMs. Which published model fits, and how many clusters remain for tenants.
Answer: Three Management Zones with Isolated Workload Zones, leaving one cluster as the workload zone. Three clusters carry the control plane VMs, one for each management zone, and the fourth becomes a workload zone assigned to vSphere Namespaces.

2. A design calls for tenants to define their own subnets and receive automatic routing and NAT without a network administrator raising a ticket. Which networking path satisfies this, and what version floor applies.
Answer: NSX VPC. Supervisor 9.0 introduced VPC network support and VKS 3.3.1 or later is required for VKS clusters to consume it. NSX segments with Tier 0 and Tier 1 give the same connectivity but the administrator still carves the segments.

3. A namespace is assigned all three vSphere Zones and given a CPU reservation of 300 MHz. A tenant complains that a single pod cannot obtain more than a third of the reservation. Is this a defect.
Answer: No. Namespace resources on a three zone Supervisor are drawn from the three underlying clusters in equal parts, so 300 MHz becomes 100 MHz per cluster. Size the reservation for per zone consumption, or assign the namespace a single zone.

Design three management zones and isolate the workload zones

My recommendation, and the one I now put in every VKS design where the host budget allows it, is Three Management Zones with Isolated Workload Zones on NSX VPC networking with Avi in the load balancing path. Three management zones because the only irreversible decision on the list deserves the safer setting. Isolated workload zones because keeping tenant pods off control plane hosts costs one cluster and removes an entire class of support conversation. NSX VPC because tenant self service is where this platform earns its keep, and Avi because Layer 7 requirements arrive eventually whether or not they were in the original brief.

Avoid Single Management Zone with Combined Workload Zones for anything you intend to keep. It is a fine shape for a lab and a bad shape for a platform, and I have never seen a team that started there stay there happily. If budget genuinely forbids twelve hosts, the honest compromise is three management zones running at the minimum three or four hosts each, with combined workload zones, and a documented plan to add an isolated workload zone later. That plan is deliverable because workload zones can be added. A plan to add management zones later is not deliverable, and writing one into a design document is how a rebuild starts.

Readers arriving from a Tanzu Kubernetes Grid Integrated estate should note that none of these zone models map cleanly onto a TKGI availability zone, and the migration path has its own series in the TKGI to VKS guide. For this exam, design from the four published models and nothing else.

Tonight, open the vSphere Client on your own lab, go to vCenter, Configure, vSphere Zones, and write down which model you are actually running. Then run kubectl get availabilityzones against the Supervisor and confirm the two agree. If your lab is a single zone, add a second zone as a workload zone and assign it to one vSphere Namespace, because doing that once is how you learn in your fingers which half of the zone story is reversible. Next part opens the Supervisor itself and works through its capabilities, services and architecture topologies against Objective 2.1.

VCAP-VKS Exam Series · Part 4 of 34
« Previous: Part 3  |  Guide  |  Next: Part 5 »

References

Supervisor Architecture and Deployment Options, Broadcom TechDocs, VCF 9.0
vSphere Supervisor Zone Models, VCF 9.0 Design Library
Create vSphere Zones for a Multi Zone Deployment with NSX, Broadcom TechDocs
Architecting VKS on VCF, Field Questions Answered, VMware Cloud Foundation blog
VMware Cloud Foundation VKS Administrator Exam Guide, 3V0-24.25

About The Author


Discover more from Journal of Intelligent Infrastructure

Subscribe to get the latest posts sent to your email.

Leave a Reply

Your email address will not be published. Required fields are marked *

Architect’s Toolkit

About the Author

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

Discover more from Journal of Intelligent Infrastructure

Subscribe now to keep reading and get access to the full archive.

Continue reading