, ,

Supervisor Cluster Enablement Process on VCF 9 (VCAP-VKS Exam Series, Part 14)

Objective 3.4 is about the process, not the wizard. Five management addresses instead of three, the decisions that lock at activation, and the enablement failures that cost a change window.

VCAP-VKS Exam Series · Part 14 of 34
Key takeaways: Objective 3.4, published wording Create the process of enabling a Supervisor cluster. Reserve five consecutive management addresses, not three: one per control plane VM, one floating address that follows the etcd leader, one buffer for replacement. Control plane size scales up only and never down, so Tiny at 2 CPUs and 8 GB is a trap in anything you intend to keep. A three zone activation turns three vSphere clusters into one Supervisor with one control plane VM per zone. Simplified activation without a load balancer gives you VM Service only, with no vSphere Pods, no Supervisor Services and no persistent volumes. Read the enablement decision lock table below as the artifact.
Who this is for: You have driven the activation wizard at least once, probably in a nested lab where DHCP handed you everything and it worked, and you have never had to explain to a change board why the Supervisor had to be torn down. This Part covers Objective 3.4, published wording Create the process of enabling a Supervisor cluster. Terms on first use: a Supervisor is the Kubernetes control plane vSphere runs on the cluster itself; a vSphere Zone is a failure domain that maps to exactly one vSphere cluster; a control plane VM is one of the appliance virtual machines that runs the Supervisor API server and etcd; etcd is the key value store holding Kubernetes state; a VPC is an NSX Virtual Private Cloud, an independent routing domain inside an NSX project; a connectivity profile is the NSX object that tells a VPC which external network and services it may reach; VM Service is the Supervisor module that runs standalone virtual machines; govc is the vSphere command line client. Where the lab stands: Parts 11 to 13 sized the load balancer and drew the address plan on paper. This Part spends it.

Enablement decisions you cannot take back

Five is the number that decides whether your activation runs clean. Not three, which is what almost everyone reserves, because a Supervisor has three control plane virtual machines and three feels like the obvious answer. Broadcom documents a block of five consecutive management addresses: one for each control plane VM, one floating address that moves to whichever control plane VM is currently the etcd leader, and one held in reserve so a failed control plane VM can be replaced without another trip to the address management team. Reserve three, and nothing rejects you at the time. You discover the gap forty minutes later when Config Status has not moved off Configuring and no error has been raised.

Objective 3.4 asks you to create the process of enabling a Supervisor cluster, and the word process is carrying the weight in that sentence. Nobody is checking whether you can find the Get Started button. What is being checked is whether you know which inputs are gathered before the wizard opens, which order they are consumed in, which of them are permanent, and what evidence proves the activation actually succeeded rather than merely finished. A candidate who has only ever clicked through a lab build tends to know the screens and none of the consequences.

Everything downstream inherits this hour. Namespaces inherit the workload network and its address pools. Storage classes inherit the policies you attach. VM classes, content libraries, Supervisor Services and every VKS cluster anyone provisions later all sit inside a boundary you drew during activation. When a workload cluster refuses to come up in Part 20 or Part 30, the honest first question is almost never about the cluster. It is about which Supervisor object was configured wrongly at activation and has been quietly wrong ever since.

One naming note, said once and then left alone. VKS is the product formerly called TKG Service or TKGS, and the rename stopped at the marketing layer. You will see tkg in resource short names, API groups and log paths throughout this procedure, and it does not mean you are looking at a deprecated product. It means the plumbing was never renamed.

DecisionWhere it is setChangeable after activationCost of changing it
Networking stack, VPC or NSX segments or vDS with AvivCenter Server and Network pageNoFull redeploy of the Supervisor
Zone topology, one zone or threeSupervisor location pageZones can be added later and assigned to namespaces, but Broadcom documents no conversion of control plane placementTreat as permanent, decide before you start
Control plane sizeAdvanced Setting pageScale up only, never downRolling replacement upward, redeploy to go smaller
Management IP block and floating IPManagement Network pageNo, changing control plane and floating IPs is not supportedFull redeploy
Load balancer choiceDetected by NSX, or Avi if installedNo, one load balancer per SupervisorFull redeploy
Storage policy for control plane VMsSupervisor location pageYes, in Supervisor configurationLow, storage migration only
API server DNS namesAdvanced Setting pageYes, certificate is regeneratedBrief API server disruption
VPC connectivity profile and private CIDRsWorkload Network pageYes, these are NSX project level objectsModerate, NSX change ticket

Enablement decision lock table. Print it, fill it in before the wizard opens, and keep it with the build record.

Preflight checks before the wizard opens

Pin your versions first, because half the study material you will find online was written against vSphere with Tanzu on vSphere 8 and the screens have moved.

# Versions this procedure was written and tested against vCenter Server 9.0.0.0 vSphere Supervisor 9.0 (embedded in vCenter 9.0) NSX 9.0 with VPC networking enabled NSX Advanced Load Balancer (Avi) in the load balancing path VKS 3.3.1 (supplies the Kubernetes releases) kubectl v1.32.x client kubectl-vsphere plugin shipped with Supervisor 9.0 govc 0.47.x

Now prove the estate rather than assuming it. Four checks catch most of what goes wrong: vCenter build, the target clusters and their zone mapping, at least one storage policy that the control plane VMs can actually land on, and forward DNS for the name you intend to use as the API server address. Read the vCenter password from an environment variable so it never lands in your shell history or a build document.

export GOVC_URL=https://vcenter01.vks.lab.internal export GOVC_USERNAME=administrator@vsphere.local export GOVC_PASSWORD="${VC_PASSWORD}" # exported by your secret tool, never typed export GOVC_INSECURE=false govc about | egrep ‘Name|Version|Build’ govc find / -type c govc storage.policy.ls dig +short sup01.vks.lab.internal # expected Name: VMware vCenter Server Version: 9.0.0 Build: 24755230 /wld01-dc/host/wld01-cl-a /wld01-dc/host/wld01-cl-b /wld01-dc/host/wld01-cl-c vSAN Default Storage Policy vks-gold-highperf vks-silver-default 10.50.12.43

That last line matters more than it looks. Login to a Supervisor by fully qualified domain name is required to avoid certificate problems, because Server Name Indication is used to present the correct certificate to incoming requests, and the certificate only carries names you declared. On my first run of this build I skipped the API Server DNS Names field on the Advanced Setting page, pointed DNS at the address anyway, and got this on the first login attempt.

$ kubectl vsphere login –server=sup01.vks.lab.internal –vsphere-username administrator@vsphere.local Error: Post "https://sup01.vks.lab.internal:443/wcp/login": tls: failed to verify certificate: x509: certificate is valid for 10.50.12.43, not sup01.vks.lab.internal

Recoverable, because API server DNS names can be edited afterwards and the certificate regenerates. It still cost an API server disruption on a cluster I had just told people was ready, which is a bad first impression to make on a platform team. Type the FQDN into the wizard.

Activation procedure with VCF Networking and VPC

VPC is the recommended workload network for a VCF stack and it is the supported network stack for VCF Automation, so it is the path this series builds on. Three prerequisites gate it: the general Supervisor cluster prerequisites are met, the Centralized Gateway is configured, and Avi is installed if you want it, because NSX falls back to the NSX Edge load balancer when it does not detect Avi.

flowchart TD
  A[Preflight, versions, zones, storage policy, DNS] --> B[vCenter Server and Network, pick VPC stack]
  B --> C[Supervisor location, zones and storage policy]
  C --> D[Management Network, five address block]
  D --> E[Workload Network, NSX project and connectivity profile]
  E --> F[Advanced Setting, control plane size and DNS names]
  F --> G[Export configuration to JSON]
  G --> H[Finish, control plane VMs deploy]
  H --> I{Config Status reaches Running}
  I -->|yes| J[Verification, login, nodes, services]
  I -->|no| K[wcpsvc log, fix, retry is automatic]
  K --> I
Activation as a state machine. Everything before Finish is a decision, everything after is a wait with evidence attached.
  1. Open Supervisor Management from the vCenter home menu and click Get Started.
  2. vCenter Server and Network page. Select the vCenter system, then select VCF Networking with VPC as the networking stack. This is the single most irreversible click in the whole wizard.
  3. Supervisor location page. Enter a Supervisor name, tick Enable control plane high availability, choose the data center holding your zones, and select a compatible vSphere Zone. Cluster Deployment places the Supervisor on one Management Zone and gives host level high availability for the control plane. A three zone deployment makes all three mapped vSphere clusters into one Supervisor and puts one control plane VM in each zone, which is cluster level high availability. Control plane HA is mandatory on a three zone activation. One caveat worth memorising: in VCF 9 you cannot select a management network per control plane node in the vSphere UI, because that requires a shared vSphere Distributed Switch, so you must use the API for that case.
  4. Storage policy. Select the policy for placement of the control plane VMs. On the reference estate this is vks-silver-default, backed by vSAN.
  5. Management Network page. Choose DHCP or Static. In DHCP mode the floating IP requires a DHCP server that supports client identifiers, because all control plane VMs use stable DHCP client identifiers. In Static mode you enter the block of five addresses, the subnet mask, the gateway, DNS servers, DNS search domains and NTP servers. Changing control plane VM addresses or the floating IP afterwards is not supported.
  6. Workload Network page. Select the NSX Project, the VPC Connectivity Profile, the Private (VPC) CIDRs from which private subnets are allocated, a DNS server and the NTP servers that NSX Manager uses. Note a VCF 9.0 behaviour change: on a new Supervisor, if the same DNS servers are shared between management and workload networks, control plane DNS lookups route through the management network, where an existing Supervisor keeps routing them through the workload network.
  7. Advanced Setting page. Set Supervisor Control Plane Size, enter the API Server DNS Names, and use Export Configuration to produce a JSON file of everything you just entered.
  8. Click Finish. Activation now creates and configures the control plane VMs and the rest of the components. Kubernetes, Velero and VM Service are enabled as part of that process.

Export Configuration is the step people skip and later wish they had not. That JSON file is your only faithful record of what was entered, it can be edited and imported to redeploy the same Supervisor, and it is the fastest way to stamp a second Supervisor with matching settings. Treat it as a build artifact and commit it next to your infrastructure code.

Write out the management block explicitly before you touch step five. Ambiguity here is what produces the forty minute silence.

# Static management block, five consecutive addresses, 10.50.12.0/24 gw 10.50.12.1 10.50.12.40 control plane VM 1 10.50.12.41 control plane VM 2 10.50.12.42 control plane VM 3 10.50.12.43 floating IP, moves to whichever VM is the etcd leader 10.50.12.44 buffer, used when a control plane VM is replaced # DNS A record must point at the floating IP, or at the load balancer VIP # fronting the Supervisor when one is present sup01.vks.lab.internal. IN A 10.50.12.43
Control plane sizeCPUsMemoryStorageWhere it fits
Tiny28 GB32 GBThrowaway demos only, you cannot shrink back to it later
Small416 GB32 GBSensible floor for a lab you will keep for the whole exam prep
Medium816 GB32 GBProduction with a moderate namespace and cluster count
Large1632 GB32 GBDense estates, many Supervisor Services and VKS clusters

Storage is 32 GB at every size, so the only levers are CPU and memory, and both move in one direction.

Verification and what green looks like

A Config Status of Running in the vCenter UI is necessary and not sufficient. Prove it from the client side, using kubectl with the vSphere plugin rather than a raw kubeconfig, because the plugin is what exchanges your vCenter Single Sign On credentials for a token and writes contexts for every namespace your account can see. That token exchange is exactly what breaks when the certificate or the identity provider is wrong, so making it the first verification step tells you two things at once.

$ kubectl vsphere login –server=sup01.vks.lab.internal –vsphere-username administrator@vsphere.local Password: Logged in successfully. $ kubectl config use-context sup01.vks.lab.internal Switched to context "sup01.vks.lab.internal". $ kubectl get nodes NAME STATUS ROLES AGE VERSION 42019f7ac1d9e2b4a06f3c5e7d81b2af Ready control-plane,master 14m v1.32.0+vmware.1 4201c3d6e8b7f95a2d40e1c86b7f30da Ready control-plane,master 13m v1.32.0+vmware.1 4201ea52b9c3418d76fa02e5d9c47b13 Ready control-plane,master 12m v1.32.0+vmware.1 esx01.wld01.lab.internal Ready agent 11m v1.32.0-sph-9b1c2d4 esx02.wld01.lab.internal Ready agent 11m v1.32.0-sph-9b1c2d4 esx03.wld01.lab.internal Ready agent 11m v1.32.0-sph-9b1c2d4 $ kubectl get pods -n vmware-system-vmop NAME READY STATUS RESTARTS AGE vmware-system-vmop-controller-manager-7c9f6bd8 2/2 Running 0 10m

Three control plane VMs with UUID style names, plus one agent node per ESX host, is what a healthy single zone activation looks like. On a three zone activation you get the same three control plane nodes, one per zone, and agent nodes from all three clusters. Watch the version strings: control plane nodes report a standard Kubernetes version with a vmware suffix, agent nodes report a Spherelet build, and a mismatch between them is a genuine signal rather than cosmetic noise.

Now the failure you should expect on a freshly activated Supervisor, and the place the tkg naming shows through.

$ kubectl get tkr No resources found # TanzuKubernetesRelease, short name tkr, is still the object name in VKS 3.3.x. # Empty here is correct and expected: activation does not associate a content # library, so there is no Kubernetes release to provision a VKS cluster from yet. # That association is namespace scoped work, covered in Part 9.

I have watched capable admins raise a support case at this exact line, convinced activation had half failed. It had not. Activation gives you a Supervisor, not a catalogue. Knowing which empty output is normal is a real part of Objective 3.4, because the process does not end at Running and it does not include content library association either.

Rollback paths and enablement failures

Rollback is blunt. Disable Supervisor from the vCenter UI is the only supported way back, and it destroys the control plane VMs, every vSphere Namespace and every workload sitting on them. Nothing about it is a partial undo. This is why the decision lock table matters more than any single command in this Part: the fallback for a wrong permanent choice is a rebuild, and the only cheap moment to catch it is before Finish.

Short of that, activation retries itself. Workload Control Plane on vCenter runs a state machine that keeps reconciling, so fixing the underlying fault often clears the condition without any operator action. Watching that state machine is far more informative than refreshing the UI.

# on the vCenter appliance, as root tail -f /var/log/vmware/wcp/wcpsvc.log | grep -Ei ‘cluster|error|retry’ # a healthy progression looks like info wcp [cluster domain-c1021] state ConfigInitializing to ConfigDeploying info wcp [cluster domain-c1021] control plane VM 1 of 3 powered on info wcp [cluster domain-c1021] state ConfigDeploying to ConfigRunning # the DHCP floating IP failure looks like this, repeating, with no fatal error warn wcp [cluster domain-c1021] floating IP not yet assigned, retrying in 60s warn wcp [cluster domain-c1021] floating IP not yet assigned, retrying in 60s
SymptomCauseRemediation
Config Status stays Configuring past 60 minutes, no error raisedControl plane VMs cannot reach vCenter on the management networkCheck routing between the Supervisor management network and vCenter, then let the state machine retry
Floating IP never assigned in DHCP modeDHCP server does not support client identifiersRedeploy with a static block of five addresses, or fix the DHCP server first
x509 SAN mismatch on kubectl vsphere loginFQDN was not entered in API Server DNS Names, so it is absent from the certificateAdd the FQDN in Supervisor configuration and let the certificate regenerate
Activation fails on capacityControl plane size larger than the cluster can placeAdd capacity, or activate smaller and scale up later, never plan to scale down
Service of type LoadBalancer stays in pending, no external addressSimplified activation with no load balancer presentAdd a load balancer to the simplified deployment, which also unlocks vSphere Pods and Supervisor Services
PersistentVolumeClaim stays Pending foreverSimplified Supervisor does not support persistent volume provisioningMove to a full activation, this is not fixable with a storage class
kubectl get tkr returns No resources foundNormal, no content library is associated yetAssociate and sync the content library, covered in Part 9
Control plane scale out unavailableSingle control plane VM in a simplified deployment without a load balancerAdd a load balancer, then scale out the control plane

Activation failure to remediation lookup. Four of these eight are consequences of choosing the simplified flow.

Activation wall clock, three attempts Measured on the reference estate, click Finish to Config Status Running 1. DHCP, one zone abandoned 2. Static block, one zone 3. Static block, three zones 51 min 26 min 38 min 0 15 30 45 60 minutes
Outlined bar is the run that never reached Running. Three zones costs roughly twelve extra minutes, not double.

Exam focus for objective 3.4

Objective 3.4, Create the process of enabling a Supervisor cluster: What this objective expects you to be able to do is assemble the activation into a correct ordered process and know which inputs are permanent. Because it is a process objective, it favours item types that test sequence and mapping rather than recall: build list, drag and drop, and matching are natural fits, with multiple selection multiple choice used for prerequisites and limitations. Expect to be asked to order the wizard stages, to match a wizard field to the object it configures, or to select every prerequisite that applies. Sizing values and limitation lists are fair game for multiple choice. The trap that catches experienced admins: assuming the management network needs three addresses because there are three control plane VMs. Five is the documented block, and the two extra addresses have named jobs, a floating address that follows the etcd leader and a replacement buffer. A close second is answering that a simplified activation can later do everything a full one can, when in fact vSphere Pods, Supervisor Services and persistent volume provisioning are absent until a load balancer is added, and persistent volumes remain out of reach on a simplified Supervisor.

Objective checkpoint

Three original questions written from the published objective wording:

1. A design calls for a Supervisor with control plane high availability on a static management network. How many management IP addresses must be reserved, and what does each one do?
Answer: Five consecutive addresses. Three for the control plane VMs, one floating address that moves to the control plane VM holding the etcd leader role, one buffer for replacing a failed control plane VM. Reasoning: Broadcom documents the static entry as a block of five, and reserving only three is the most common preflight error on this objective.

2. An administrator activated a Supervisor at Tiny control plane size to conserve lab capacity. Six weeks later the estate has grown and the API server is under pressure. What are the options?
Answer: Scale up to Small, Medium or Large. Scaling back down later is not possible without redeploying the Supervisor. Reasoning: Control plane size scales up only, which makes the initial choice a one way decision even though it appears on a page labelled Advanced Setting.

3. A team used the simplified deployment flow with no load balancer to stand up a Supervisor quickly. They now report that a PersistentVolumeClaim stays Pending and a LoadBalancer Service has no external address. Which of these is fixed by adding a load balancer?
Answer: Only the LoadBalancer Service. Persistent volume provisioning is not supported on a simplified Supervisor at all. Reasoning: Adding a load balancer enables vSphere Pods, Supervisor Services and control plane scale out, but persistent volume provisioning is listed separately as unsupported in that configuration.

Activation call for this estate

Field note, and it is the reason the chart above has an outlined bar. On a customer build I chose DHCP for the management network, because their DHCP estate was well run and the alternative meant waiting two days for a static reservation. Control plane VMs picked up addresses in about nine minutes and I relaxed. Config Status then sat at Configuring for fifty one minutes with no error anywhere in the UI, while wcpsvc.log repeated a single warning about the floating IP every sixty seconds. Their DHCP server did not honour client identifiers. We tore the whole thing down, waited for the static block, and the second attempt reached Running in twenty six minutes. Total loss was two hours and ten minutes plus a change window I could not get back, over a requirement that is one sentence in the documentation.

My pick for the reference estate is a three zone activation on VCF Networking with VPC, control plane HA on, Small control plane size, and a static block of five management addresses with the API server FQDN entered on the Advanced Setting page. Three zones costs about twelve extra minutes of wall clock and buys cluster level high availability for the control plane, which is the cheapest resilience you will ever purchase on this platform. What I would avoid is the advice you will read everywhere, that the simplified deployment flow is a fine way to get moving because you can add a load balancer afterwards. For a study estate it is actively harmful. It gives you VM Service and nothing else, which means vSphere Pods, Supervisor Services and persistent volumes, the subject matter of Objectives 4.3, 4.4 and 4.11, simply do not exist on it. You will build muscle memory for a Supervisor that is not the one being examined. Build the full activation, even if it takes an afternoon.

What a clean result looks like: Config Status shows Running in vCenter. kubectl vsphere login succeeds by FQDN with certificate verification on, not skipped. kubectl get nodes lists three control plane nodes plus one agent node per ESX host, all Ready. Control plane pods in vmware-system-vmop and the other system namespaces are Running with no restarts. Your exported configuration JSON is committed alongside your infrastructure code. Your decision lock table is filled in and attached to the build record. kubectl get tkr is empty, and you know that is correct.

Tonight, in your own lab, do one thing: open Advanced Setting on your existing Supervisor and write down its control plane size, then work out what you would do if it needed to be smaller. If the honest answer is redeploy, you have understood this objective. Readers who want the product level walkthrough beneath the exam view should keep the VKS complete guide open alongside this series, the networking objects behind the VPC fields are covered in the NSX complete guide, and the load balancer sizing that feeds step two lives in Part 11. Part 15 turns to service mesh design on VKS, the last stop in Section 3 before the hands on run of Section 4 begins.

VCAP-VKS Exam Series · Part 14 of 34
« Previous: Part 13  |  Guide  |  Next: Part 15 »

References

About The Author


Discover more from Journal of Intelligent Infrastructure

Subscribe to get the latest posts sent to your email.

Leave a Reply

Your email address will not be published. Required fields are marked *

Architect’s Toolkit

About the Author

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

Discover more from Journal of Intelligent Infrastructure

Subscribe now to keep reading and get access to the full archive.

Continue reading