, ,

VKS Cluster Provisioning, Monitoring and Scaling Commands (VCAP-VKS Exam Series, Part 20)

Objective 4.5 asks you to provision, delete, monitor and scale VKS clusters with kubectl and the VCF CLI. Here is the full lifecycle on a VCF 9.0 estate, with the operations the Supervisor refuses and the one the VCF CLI cannot do at all.

VCAP-VKS Exam Series · Part 20 of 34

A candidate on a study call asked me which command line the exam expects for scaling a VKS cluster, kubectl or the VCF CLI. That is the wrong question, and understanding why it is wrong is most of objective 4.5. Both tools write the same field on the same object. What separates a pass from a fail is knowing which changes to that field the platform accepts, which ones it refuses outright, and which scaling operation the VCF CLI cannot express at all.

Key takeaways: Objective 4.5 covers four verbs, provision, delete, monitor and scale, across two tools that both drive Cluster API underneath. Horizontal scaling changes a replicas count and completes in single digit minutes. Vertical scaling changes a VM class and triggers a full rolling replacement, which cost me 38 minutes on three workers where the horizontal equivalent cost 9. Control plane replicas may be 1 or 3, scale out is supported and scale in is not. And the headline command to keep in your fingers is vcf cluster get CLUSTER –show-all-conditions all, because plain get hides the condition that tells you why nothing is happening.
Who this is for: A candidate who has a Supervisor activated, namespaces carved and add-on services running, and now needs to run workload clusters through their whole life from a terminal. This Part covers Objective 4.5, published wording Provision, delete, monitor and scale VKS clusters using kubectl and VCF CLI tools. Terms defined on first use here: VKS is vSphere Kubernetes Service, the product formerly named TKG Service or TKGS, so tkg keeps appearing in API groups, resource kinds and documentation paths and that rename is not finished; a Supervisor is the Kubernetes control plane running on ESX hosts that owns namespaces and provisions clusters; a vSphere Namespace is the tenancy and quota boundary a cluster lives inside; a VM Class is the CPU and memory shape a node virtual machine takes; a VKr is a vSphere Kubernetes release, the versioned node image and Kubernetes version pair, formerly a TKr; Cluster API, abbreviated CAPI, is the upstream project whose Cluster, MachineDeployment and Machine objects VKS uses; a ClusterClass is the reusable topology template a Cluster references; VCF CLI is the vcf command that replaced the Tanzu CLI.

Last part we installed Harbor as a Supervisor Service and external-dns as a standard package inside a workload cluster. This part we go one layer down and drive a cluster called vks-lifecycle-01 through provisioning, monitoring, both kinds of scaling and deletion, inside the vSphere Namespace ns-platform-a on the three zone estate this series has been building since Part 14. Everything below assumes that namespace already has a storage policy, a VM class and a content library bound to it, because every failure in this Part traces back to one of those three Supervisor objects.

Preflight for cluster lifecycle work

Two authentication paths exist and they are not interchangeable. The kubectl-vsphere plugin merges Supervisor and namespace contexts into your kubeconfig and is what almost every Broadcom procedure assumes. VCF CLI keeps its own context store and needs one created before any cluster subcommand will resolve. Candidates who only ever use one of them lose points on items that name the other, so set both up in your lab tonight and keep both in the shell history.

# Versions this Part was written and tested against # VCF 9.0, vSphere Supervisor 9.0, VKS 3.3.1 # VKr v1.32.x, kubectl v1.32.3, VCF CLI v9.0.x # kubectl-vsphere plugin as shipped by vCenter 9.0 # Path A, kubectl plugin. Password is read from an env var, never inline. export KUBECTL_VSPHERE_PASSWORD="$(read -rsp ‘SSO password: ‘ p; echo "$p")" kubectl vsphere login –server sup-a.lab.local –vsphere-username platform-admin@lab.local –tanzu-kubernetes-cluster-namespace ns-platform-a Logged in successfully. You have access to the following contexts: sup-a.lab.local ns-platform-a kubectl config use-context ns-platform-a # Path B, VCF CLI context. Note the flag is –ca-certificate, not –cacert. vcf context create –endpoint sup-a.lab.local –username platform-admin@lab.local –ca-certificate ~/sup-a-ca.pem vcf context list NAME TYPE ENDPOINT ACTIVE sup-a kubernetes sup-a.lab.local true

Three preflight checks prove the namespace can actually build a cluster, and running them takes under a minute. Skipping them is how people end up staring at a Machine stuck in Provisioning for half an hour.

kubectl get virtualmachineclass -n ns-platform-a NAME CPU MEMORY AGE best-effort-medium 2 8Gi 21d guaranteed-medium 2 8Gi 21d guaranteed-large 4 16Gi 6d kubectl get storageclass NAME PROVISIONER RECLAIMPOLICY AGE vks-default csi.vsphere.vmware.com Delete 21d vks-high-performance csi.vsphere.vmware.com Delete 21d kubectl get vkr NAME VERSION READY COMPATIBLE v1.31.4—vmware.1-fips-vkr.1 v1.31.4+vmware.1 True True v1.32.0—vmware.2-fips-vkr.1 v1.32.0+vmware.2 True True

If the VKr list comes back empty, stop. That is a content library that has not synced, not a cluster problem, and no amount of retrying the manifest will fix it. If COMPATIBLE reads False on the release you want, your Supervisor version and that VKr do not pair, and provisioning will be rejected by the admission webhook rather than failing halfway.

Provisioning paths, kubectl and VCF CLI

Here is the artifact to keep from this Part. Print the table, cover the right hand columns, and work across it until you can produce either command from memory. Objective 4.5 names both tools explicitly, so an item is free to show you one and ask for the equivalent in the other. Notice the last column, because it is the point: every row lands on the same Cluster object and the same field. Nothing in the VCF CLI is a separate control plane.

OperationkubectlVCF CLIWhat it writes
Provisionkubectl apply -f cluster.yamlvcf cluster create -f cluster.yamlCreates the Cluster object in the namespace
Listkubectl get cluster -n ns-platform-avcf cluster list -n ns-platform-aRead only, no write
Inspect deeplykubectl describe cluster NAMEvcf cluster get NAME –show-all-conditions allRead only, renders the CAPI object tree
Scale workerskubectl edit cluster/NAMEvcf cluster scale NAME -w 6 -p node-pool-1spec.topology.workers.machineDeployments[].replicas
Scale control planekubectl edit cluster/NAMEvcf cluster scale NAME -c 3spec.topology.controlPlane.replicas
Scale verticallykubectl edit cluster/NAMEnot availablespec.topology.variables vmClass, triggers rolling update
Fetch kubeconfigkubectl vsphere login –tanzu-kubernetes-cluster-name NAMEvcf cluster kubeconfig get NAMEMerges a context into your kubeconfig
Deletekubectl delete cluster NAMEvcf cluster delete NAME -yDeletes the Cluster, CAPI cascades to Machines
Collect logsno direct equivalentvcf cluster support-bundler createRead only, writes a bundle to disk

Two rows deserve a second look. Vertical scaling has no VCF CLI verb, which is the single most common wrong assumption I hear from candidates who learned the tool before reading the objective. And support bundle collection has no kubectl equivalent, so an item asking how you gather node level logs for a wedged cluster has exactly one right answer.

Now provision. A v1beta1 Cluster referencing the shipped ClusterClass is the shape you should be able to write from a blank file, because build list and hot area items lean on knowing where each key sits.

apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: name: vks-lifecycle-01 namespace: ns-platform-a spec: clusterNetwork: services: cidrBlocks: ["10.96.0.0/12"] pods: cidrBlocks: ["192.168.0.0/16"] serviceDomain: cluster.local topology: class: tanzukubernetescluster version: v1.32.0—vmware.2-fips-vkr.1 controlPlane: replicas: 1 workers: machineDeployments: – class: node-pool name: node-pool-1 replicas: 3 variables: – name: vmClass value: guaranteed-medium – name: storageClass value: vks-default # Apply it with either tool. Same result, same object. kubectl apply -f vks-lifecycle-01.yaml cluster.cluster.x-k8s.io/vks-lifecycle-01 created # or vcf cluster create -f vks-lifecycle-01.yaml -y
flowchart TD
  A[kubectl apply or vcf cluster create] --> B[Cluster object in vSphere Namespace]
  B --> C[ClusterClass expands the topology]
  C --> D[KubeadmControlPlane and MachineDeployment]
  D --> E[Machine objects, one per node]
  E --> F[VM Service creates VirtualMachine]
  F --> G[VM Class bound to namespace supplies CPU and memory]
  F --> H[Storage policy supplies the node disk]
  F --> I[Content library supplies the VKr image]
  G --> J[Node joins, Machine goes Running]
  H --> J
  I --> J
Every provisioning failure lands on one of the three Supervisor objects on the bottom row, not on the manifest you typed.

Monitoring commands that report real state

Plain get is a summary and it lies by omission. A cluster can sit at Provisioned with a Machine that has been stuck for twenty minutes and the top line will not say so. Work down the object tree instead, from Cluster to MachineDeployment to Machine, and read conditions rather than phases.

kubectl get cluster -n ns-platform-a NAME PHASE AGE VERSION vks-lifecycle-01 Provisioned 14m v1.32.0+vmware.2 kubectl get machine -n ns-platform-a NAME CLUSTER PHASE AGE vks-lifecycle-01-cp-4kx2r vks-lifecycle-01 Running 14m vks-lifecycle-01-node-pool-1-w7q9d vks-lifecycle-01 Running 12m vks-lifecycle-01-node-pool-1-x2m4t vks-lifecycle-01 Running 12m vks-lifecycle-01-node-pool-1-z8p6h vks-lifecycle-01 Provisioning 12m # One Machine has been Provisioning for twelve minutes. Ask why. kubectl describe machine vks-lifecycle-01-node-pool-1-z8p6h -n ns-platform-a Events: Warning FailedCreate 11m vspheremachine-controller admission webhook "default.validating.virtualmachine.vmoperator.vmware.com" denied the request: limit is exceeded, memory limit 128Gi, requested 144Gi in namespace ns-platform-a

That is the failure I want you to internalise, because it is not a Kubernetes problem at all. Nothing is wrong with the manifest, the VKr or the VM class. A namespace memory limit set months earlier by whoever carved ns-platform-a is refusing the fourth node. Raise the limit on the vSphere Namespace and the Machine proceeds without any change to the cluster spec. Objective 5.1 revisits this family of errors and Part 30 goes deeper, but you should already be tracing failures upward to the Supervisor by now.

# VCF CLI renders the same tree with conditions attached. vcf cluster get vks-lifecycle-01 -n ns-platform-a –show-all-conditions all NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES vks-lifecycle-01 ns-platform-a running 1/1 3/4 v1.32.0+vmware.2 Details: NAME READY SEVERITY REASON SINCE Cluster/vks-lifecycle-01 False Warning ScalingUp 12m ClusterInfrastructure True 14m ControlPlane/vks-lifecycle-01-cp True 14m Workers MachineDeployment/node-pool-1 False Warning WaitingForMachine 12m # Also worth knowing, and easy to get wrong under time pressure: # –show-v1beta2 only takes effect alongside –show-all-conditions. vcf cluster list -A -o table
Monitoring habit worth building: get the phase, then immediately get the Machines. Phase answers what the cluster thinks it is. Machines answer what the estate actually built. On a three zone Supervisor those two disagree more often than you would expect, because zone placement is decided per Machine and a zone short on capacity will hold exactly one node hostage while the rest come up clean.

Scaling operations and what each one costs

Four scaling axes exist and they are not equally permitted. Broadcom publishes the matrix and it is short enough to memorise, which makes it prime matching item material.

Node roleScale outScale inVerticalVolume
Control planeYes, 1 to 3NoYesYes, vSphere 8 U3 or later only
WorkerYesYesYesYes, data in the current volume is lost on the roll

Control plane replicas must be odd, so 1 or 3, and nothing else. Scale out is supported. Scale in is not, and that asymmetry catches people who assume symmetry because worker pools have it. Run the refusal in your own lab once so the error text is familiar.

# Horizontal, workers, 3 to 6. VCF CLI needs the pool name on a multi pool cluster. vcf cluster scale vks-lifecycle-01 -n ns-platform-a -w 6 -p node-pool-1 Updating workload cluster ‘vks-lifecycle-01’… Successfully updated the workload cluster. # Same change via kubectl, which is the path the exam guide procedures use. kubectl edit cluster/vks-lifecycle-01 -n ns-platform-a # spec.topology.workers.machineDeployments[0].replicas: 3 -> 6 cluster.cluster.x-k8s.io/vks-lifecycle-01 edited # Control plane 1 to 3, supported. vcf cluster scale vks-lifecycle-01 -n ns-platform-a -c 3 Successfully updated the workload cluster. # Control plane 3 back to 1, refused. This is the one to have seen before. vcf cluster scale vks-lifecycle-01 -n ns-platform-a -c 1 Error: unable to scale down control plane nodes; scaling in the control plane is not supported for VKS clusters

Vertical scaling is a different animal wearing the same word. Changing the vmClass variable does not resize anything in place. VKS rolls out replacement nodes on the new class and drains the old ones, which is a full rolling update with all of its usual hazards, including PodDisruptionBudgets that can stall it indefinitely. There is no VCF CLI verb for it.

kubectl edit cluster/vks-lifecycle-01 -n ns-platform-a # variables: # – name: vmClass # value: guaranteed-medium -> guaranteed-large # Watch the replacement, do not watch the phase. kubectl get machine -n ns-platform-a -w NAME PHASE AGE vks-lifecycle-01-node-pool-1-w7q9d Running 3h12m vks-lifecycle-01-node-pool-1-x2m4t Deleting 3h12m vks-lifecycle-01-node-pool-1-b4n7k Provisioning 40s # A stalled roll almost always looks like this on the drained node. kubectl describe machine vks-lifecycle-01-node-pool-1-x2m4t -n ns-platform-a Warning DrainFailed 9m Drain not completed yet: cannot evict pod as it would violate the pod disruption budget pdb: pg-primary-pdb, namespace: data
Wall clock cost of each scaling axis Minutes from command to all Machines Running. Three zone VCF 9.0 estate, VKS 3.3.1, guaranteed-medium nodes, averaged over three runs. Workers plus 1 7 min Workers plus 3 9 min Workers minus 3 4 min Control plane 1 to 3 14 min VM class, 3 workers 38 min Vertical scaling of three workers cost more than four times the horizontal equivalent, because every node is replaced rather than resized.

Read the bottom bar and then read the second bar again. Adding three nodes took 9 minutes. Making three existing nodes bigger took 38. Broadcom says plainly in the scaling documentation that horizontal scaling is generally the preferred approach, and the measurement is why. Most study material treats the two as interchangeable ways to add capacity. They are not, and a design item that gives you a maintenance window and asks which approach fits is testing exactly this gap.

Deletion, rollback and failure signatures

Rollback for a scale operation is simply the inverse edit, with one exception that matters. Horizontal changes reverse cleanly because replicas is just a number. Vertical changes do not reverse cheaply, because setting vmClass back triggers a second full rolling update at the same 38 minute cost, so you pay twice for the round trip. Volume changes on worker nodes do not reverse at all in any meaningful sense, since the roll discards the data in the existing volume.

# Delete. Either tool, cascading through CAPI to Machines and then VirtualMachines. vcf cluster delete vks-lifecycle-01 -n ns-platform-a -y Deleting workload cluster ‘vks-lifecycle-01’… # Verify the cascade actually finished rather than trusting the exit code. kubectl get machine,virtualmachine -n ns-platform-a No resources found in ns-platform-a namespace. kubectl get pvc -n ns-platform-a NAME STATUS VOLUME CAPACITY pg-data-vks-lifecycle-01 Bound pvc-9c1 40Gi # Persistent volumes provisioned by workloads are NOT removed with the cluster. # That leftover PVC still counts against the namespace storage quota.

Leftover persistent volume claims after a cluster delete are the quiet cost of a lab that gets rebuilt often. Mine accumulated 340 GB of orphaned claims across six rebuilds before a storage quota rejection made me look. Check pvc and cnsvolume state after every teardown.

What you seeCauseRemediation
kubectl get vkr returns nothingContent library has not synced to the namespaceResync the library in vCenter, confirm it is added to the vSphere Namespace
Machine stuck Provisioning, webhook denied, limit is exceededvSphere Namespace CPU or memory limit reachedRaise the namespace limit in vCenter, no cluster edit needed
Machine stuck Provisioning, no VirtualMachineClass foundVM class named in the manifest is not bound to the namespaceAdd the class to the namespace, or edit the manifest to a bound class
Scale to -c 1 rejectedControl plane scale in is not supportedRebuild the cluster if you truly need a single node control plane
Rolling update halted, DrainFailed on one MachineA PodDisruptionBudget forbids the last evictionRelax the budget or scale the workload out first, then let the roll continue
updates to immutable fields are not allowedControl plane volume change on a build older than vSphere 8 U3Declare the volume at creation time, it cannot be added later on that build
New workers larger than existing workersVM class definition edited after cluster creationForce a full rolling update so every node picks up the current definition

Exam focus for objective 4.5

Exam focus, Objective 4.5: Published wording is Provision, delete, monitor and scale VKS clusters using kubectl and VCF CLI tools, and the phrase that carries the weight is and, not or. Both tool families are in scope and an item can show you one and require the other. Expect matching items pairing a scaling axis with whether it is supported for a given node role, build list items sequencing a provisioning or teardown workflow, multiple selection items on which flags a VCF CLI subcommand accepts, and point and click or hot area items on where a value sits inside a Cluster manifest topology block. Trap that catches experienced admins: treating horizontal and vertical scaling as two spellings of the same operation. Horizontal edits a count and finishes in minutes; vertical replaces every node through a rolling update, has no VCF CLI verb, and can stall on a PodDisruptionBudget. A second trap sits in control plane symmetry, because scale out to three is supported and scale in back to one is not, and the answer to a scenario that needs one control plane node is a rebuild rather than a scale operation.

Objective checkpoint

Question 1. A production cluster runs three control plane nodes and six workers. A capacity review asks you to return the control plane to a single node to reclaim resources. What do you do?
Answer: tell them it cannot be done as a scale operation and requires a new cluster. Reasoning: control plane scale out is supported and scale in is not, so 3 to 1 is rejected by the platform regardless of which tool issues it.

Question 2. A cluster needs its worker nodes moved from guaranteed-medium to guaranteed-large inside a two hour window. Which tool performs it and what is the main risk to the window?
Answer: kubectl edit on the Cluster object, because the VCF CLI exposes no vertical scaling verb, and the risk is a rolling replacement stalling on a PodDisruptionBudget. Reasoning: changing vmClass rolls out new nodes and drains old ones, so eviction policy on the workloads governs whether the change completes.

Question 3. Immediately after a worker scale out, one Machine sits in Provisioning while the others reach Running. Which Supervisor object should you inspect first?
Answer: the vSphere Namespace, specifically its CPU and memory limits and its VM class bindings. Reasoning: the other Machines proving out means the manifest, the VKr and the ClusterClass are all fine, which leaves namespace level capacity or binding as the cause.

Lifecycle CLI call for this estate

My war story for this objective cost a customer ninety minutes and it started with a helpful change. Someone raised the memory on the guaranteed-medium VM class in vCenter from 8 GB to 32 GB, reasonably enough, because a different team needed bigger virtual machines. Nobody told me. Two weeks later I scaled a six worker cluster to nine, and the three new workers came up with 32 GB while the six existing ones stayed at 8. VM classes are not immutable, and Broadcom documents this exact outcome: new nodes use the updated definition, existing nodes keep the definition they were built with. The scheduler did what schedulers do and packed the fat nodes. When one of them was drained for patching a fortnight later, the workload had nowhere to land and a customer facing service degraded for eleven minutes. Reconciling it meant a full rolling update of all nine nodes, ninety minutes, in a change window I had to beg for.

So here is my call for this estate, and it has two halves. For daily work use kubectl as the primary tool and treat the VCF CLI as the reporting and support layer. Kubectl covers every write operation including vertical scaling, its output is the same output every Broadcom procedure shows you, and it does not hide a topology behind a friendly flag. Reach for the VCF CLI for three things specifically: vcf cluster get with all conditions when something is stuck, vcf cluster list -A when you need a fleet view across namespaces, and vcf cluster support-bundler create when you need node level logs, which kubectl cannot give you. The tool to avoid as a default is vcf cluster scale, not because it is broken but because it teaches the wrong mental model: it makes scaling feel like a command you run rather than a field you set, and that misunderstanding is precisely what objective 4.5 items are built to detect.

Second half: put a change control fence around VM class definitions. Treat them as shared platform contracts, not per team knobs. If a team needs a different shape, add a class rather than editing one. That single policy would have prevented the whole ninety minute episode.

Clean result checklist: Cluster phase reads Provisioned and every Machine reads Running. Worker count in kubectl matches the replicas you set in spec.topology. Control plane replicas is 1 or 3 and nothing else. A vertical change has replaced every node, verified by comparing Machine ages rather than by trusting the cluster phase. Both a kubectl context and a VCF CLI context resolve against the same cluster. After a delete, kubectl get machine and kubectl get virtualmachine both return nothing in the namespace, and you have checked for orphaned persistent volume claims.

Tonight, in your own lab: provision vks-lifecycle-01, scale the workers out by three, then change the VM class and time both operations with a stopwatch. Feeling the difference between 9 minutes and 38 is worth more than reading it here. If the underlying product mechanics are still fuzzy, the VKS Series walks the same ground without the exam framing, and the VCAP-VKS exam guide maps every objective to its Part. Part 21 picks up where a vertical scale leaves you, inside a rolling update, and covers objective 4.6.

VCAP-VKS Exam Series · Part 20 of 34
« Previous: Part 19  |  Guide  |  Next: Part 21 »

References

About The Author


Discover more from Journal of Intelligent Infrastructure

Subscribe to get the latest posts sent to your email.

Leave a Reply

Your email address will not be published. Required fields are marked *

Architect’s Toolkit

About the Author

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

Discover more from Journal of Intelligent Infrastructure

Subscribe now to keep reading and get access to the full archive.

Continue reading