, ,

Kubernetes Architecture, Networking, Storage, Service Mesh and Helm (VCAP-VKS Exam Series, Part 3)

Objective 1.2 does not test Kubernetes trivia. It tests whether you can look at a VKS cluster and name the vSphere object that created what you are seeing. Four kubectl commands, two real failures, and the inheritance map that makes the rest of the blueprint readable.

VCAP-VKS Exam Series · Part 3 of 34

Run one command against a freshly provisioned VKS cluster in my lab and four storage classes come back, even though the vSphere administrator only ever created two storage policies. Nobody typed those four names into a manifest, and working out where they came from is most of what Objective 1.2 is actually asking you to do.

$ kubectl get storageclass NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION vsan-default-storage-policy csi.vsphere.vmware.com Delete Immediate true vsan-default-storage-policy-lb csi.vsphere.vmware.com Delete WaitForFirstConsumer true vks-high-performance csi.vsphere.vmware.com Delete Immediate true vks-high-performance-lb csi.vsphere.vmware.com Delete WaitForFirstConsumer true
Key takeaways:
• Objective 1.2 is published as Use Kubernetes architecture, networking, storage, service mesh, and Helm. Note the verb. Broadcom wrote use, not describe.
• Almost nothing inside a VKS cluster is self created. Storage classes, node placement, the load balancer VIP and the machine images all descend from a vSphere Namespace on the Supervisor.
• VKS clusters support exactly two CNIs: Antrea, which is the default and needs no declaration, and Calico, which you select in the cluster YAML or by changing the Supervisor default.
• Persistent volumes travel through pvCSI in the cluster, CNS CSI on the Supervisor and CNS in vCenter. Three components, and the reader who cannot name them cannot troubleshoot Section 5.
• Helm and Istio both run on VKS, but neither is installed for you. Standard packages arrive through a repository you enable, and that is a later objective, not this one.
Who this is for: You picked a compute model in Part 2 and now you need a cluster you can read. I assume kubectl is installed, that you have logged in once with the vSphere plugin, and that general Kubernetes teaching belongs somewhere else. Component mechanics live in the vSphere Kubernetes Service complete guide. This part is about reading a cluster back to its source.

Candidates preparing for 3V0-24.25 keep reaching for CKA material when they hit this objective. I understand the instinct and I think it wastes a fortnight. On a VKS cluster you will never run kubeadm, never place a static pod manifest, never take an etcd snapshot by hand and never rotate a control plane certificate with kubeadm certs renew. Those tasks belong to Cluster API controllers running on the Supervisor, and Broadcom did not write a single objective about them. What Objective 1.2 wants is narrower and, in my experience coaching people through this exam, much harder: given a Kubernetes object in front of you, name the vSphere object that produced it.

One naming note that will save you confusion in every command below. VKS, vSphere Kubernetes Service, is the product formerly called TKG Service or TKGS. Broadcom renamed the product but not the plumbing, so API groups, CRDs, namespaces such as tkg-system and half the documentation URLs still say tkg. Read tkg as VKS and keep going. TKGS is not a separate live product you also need to learn.

Control plane anatomy from a kubectl session

Last part we compared vSphere Pods, VM Service machines and VKS clusters on paper. This part we open a terminal against the cluster that came out of that comparison, a three node VKS cluster called vks-prod-01 sitting in a vSphere Namespace called vks-prod on our three zone estate. Everything that follows was run against these versions, and I would not trust output from an older Supervisor to match.

# Estate for every command in this part # VCF 9.0, vSphere Supervisor 9.0, NSX with VPC networking, Avi in the load balancing path # VKS 3.3.1, which tops out at VKr 1.32, vSAN backing two storage policies # Credentials come from the environment, never from the command line history export VSPHERE_USER="admin@vsphere.local" export KUBECTL_VSPHERE_PASSWORD="$(read -rsp ‘password: ‘ p; echo "$p")" kubectl vsphere login –server=supervisor.lab.local –vsphere-username "$VSPHERE_USER" –insecure-skip-tls-verify –tanzu-kubernetes-cluster-namespace vks-prod –tanzu-kubernetes-cluster-name vks-prod-01

Two things deserve a comment. I use kubectl with the vSphere plugin rather than a raw kubeconfig because that plugin is what wires your vCenter identity into both API servers at once, and because every identity objective later in the blueprint assumes it. And –insecure-skip-tls-verify is a lab habit, not a production one. Objective 2.5 in Section 2 is partly about the certificate work that lets you drop that flag.

A successful login leaves you holding two contexts, and confusing them is the single most common self inflicted wound I see. One context points at the Supervisor, where a vSphere Namespace behaves like a Kubernetes namespace and where Cluster API objects live. Another points at the VKS cluster, where you are a normal Kubernetes user with real cluster admin. Same kubectl binary, two completely different object catalogues.

# Context one, the Supervisor. Cluster API objects, no application pods. $ kubectl config use-context vks-prod $ kubectl get cluster,machine -n vks-prod NAME PHASE AGE VERSION cluster.cluster.x-k8s.io/vks-prod-01 Provisioned 9d v1.32.0+vmware.1 NAME CLUSTER NODENAME PHASE AGE machine.cluster.x-k8s.io/vks-prod-01-cp-8s4vq vks-prod-01 vks-prod-01-cp-8s4vq Running 9d machine.cluster.x-k8s.io/vks-prod-01-np1-h2xkd vks-prod-01 vks-prod-01-np1-h2xkd Running 9d machine.cluster.x-k8s.io/vks-prod-01-np1-r9tlm vks-prod-01 vks-prod-01-np1-r9tlm Running 9d # Context two, inside the cluster. Nodes, not machines. $ kubectl config use-context vks-prod-01 $ kubectl get nodes -o wide –label-columns topology.kubernetes.io/zone NAME STATUS ROLES VERSION ZONE vks-prod-01-cp-8s4vq Ready control-plane v1.32.0+vmware.1 zone-a vks-prod-01-np1-h2xkd Ready <none> v1.32.0+vmware.1 zone-b vks-prod-01-np1-r9tlm Ready <none> v1.32.0+vmware.1 zone-c

Read that last column carefully, because it is the whole third premise of this series in one field. Kubernetes believes it discovered those zone labels. It did not. A vSphere administrator mapped three vSphere Zones to three vSphere clusters, attached them to the Supervisor, and the Cluster API provider stamped the labels onto nodes it created. Delete a zone in vCenter and your Kubernetes topology constraints stop working, with no Kubernetes level explanation of why.

Machine objects on the Supervisor and Node objects in the cluster describe the same virtual machines from two sides. When a node goes NotReady, the answer is almost never in the cluster. It is in the Machine object, the VirtualMachine object underneath it, or the vSphere task list. Practise pivoting between the two contexts now and Section 5 becomes far less frightening.

Pod networking, services and where a VIP comes from

A CNI, Container Network Interface, is the plugin that gives pods addresses and enforces network policy. VKS clusters support two of them and no others: Antrea, which uses Open vSwitch, and Calico, which uses the Linux bridge with BGP and enforces policy through IP tables. Antrea is the system default, so a cluster manifest that says nothing about a CNI gets Antrea. Calico is selected either in the cluster YAML or by changing the Supervisor wide default. There is also an Antrea NSX Routed variant for routable pod networking, which matters when something outside the cluster needs to address a pod directly.

What trips people up is that the CNI stops at the edge of the cluster. Pod to pod traffic and network policy belong to Antrea or Calico. ClusterIP and NodePort services belong to Antrea or Calico. A Service of type LoadBalancer belongs to none of them. That address is minted by whichever load balancer the Supervisor was enabled with, which on our estate is Avi, and on other estates is the NSX load balancer or the Foundation Load Balancer. Ingress belongs to nobody at all until you install a third party ingress controller such as Contour, because VKS does not ship one and does not pretend to.

EndpointWho provides itWhat breaks it
Pod connectivityAntrea or CalicoCNI pods failing to pull their image on an air gapped cluster
ClusterIPAntrea or CalicoService CIDR overlap declared at cluster creation
NodePortAntrea or Calico, via kube proxyNSX or VPC rules blocking the node port range
LoadBalancerAvi, NSX load balancer or Foundation Load BalancerVIP pool exhaustion on the Supervisor, not a cluster fault
IngressA third party controller you installExpecting one to exist by default
Network policyAntrea via Open vSwitch, Calico via IP tablesAssuming NSX distributed firewall rules apply to pods

Prove the split to yourself in ninety seconds. Expose the stateless web deployment we have been carrying since the start of the series and watch which component answers.

$ kubectl get pods -n kube-system -l app=antrea -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName NAME NODE antrea-agent-4kz9d vks-prod-01-cp-8s4vq antrea-agent-nn7cx vks-prod-01-np1-h2xkd antrea-agent-w2ppl vks-prod-01-np1-r9tlm antrea-controller-6b9d7f4c5d-tq8vn vks-prod-01-np1-h2xkd $ kubectl expose deployment web –type=LoadBalancer –port=80 –name=web-lb service/web-lb exposed $ kubectl get svc web-lb -w NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE web-lb LoadBalancer 10.96.142.201 <pending> 80:31688/TCP 3s web-lb LoadBalancer 10.96.142.201 192.168.140.24 80:31688/TCP 22s

Twenty two seconds of pending is normal and it is not the cluster thinking. That gap is Avi being asked for a virtual IP through the Supervisor. If EXTERNAL-IP never leaves pending, stop looking at Antrea logs. Look at the load balancer, which is exactly what Objective 3.1 is about later in the blueprint.

Scope note: NSX itself is a large subject and this series does not re-teach it. Segments, VPCs, tier gateways and distributed firewall behaviour are covered in the NSX complete guide. Here you only need to know which layer owns which endpoint in the table above.

Storage classes inherited from a vSphere Namespace

Back to those four storage classes. A vSphere administrator writes storage policies in vCenter, which are rules describing what a datastore must offer. Assigning a policy to a vSphere Namespace does two things at once: it decides which datastores that namespace may touch, and it causes the Supervisor to generate a matching Kubernetes storage class. That storage class is then propagated down into every VKS cluster provisioned in the namespace. Two policies became four classes because each one appears in two editions, one with Immediate volume binding and one with WaitForFirstConsumer.

Names vary by estate, so read yours rather than copying mine. What does not vary is the pairing, and choosing between the two editions has a measurable cost that nobody mentions in training. I ran the same 20Gi ReadWriteOnce claim ten times against each of our four classes and timed from apply to Bound.

PVC bind time by storage class editionMedian of 10 runs, 20Gi ReadWriteOnce claim on vSAN, three zone VKS 3.3.1 labDefault policy, Immediate4 sHigh performance, Immediate6 sDefault policy, WaitForFirstConsumer38 sHigh performance, WaitForFirstConsumer41 sLate binding waits for the pod to be scheduled before a disk is created anywhere
Roughly 34 seconds of extra latency buys correct zone placement. On a single zone estate it buys nothing.

That is a real trade off and it has a clear answer. On a multi zone estate, use the WaitForFirstConsumer edition for anything with a pod affinity or topology constraint, because Immediate binding will happily create a disk in zone A and then leave your pod unschedulable in zone C. On a single zone estate, Immediate binding is faster and costs you nothing. I have watched teams standardise on late binding everywhere as a supposed best practice and then spend a sprint wondering why their CI pipeline got slower.

Knowing why the wait exists means knowing the path a claim travels. Nothing is provisioned inside your cluster. pvCSI, the paravirtual CSI driver that lives in the VKS cluster, has no vCenter credentials at all and cannot create a disk. It hands the request up.

flowchart LR
  A[PVC applied in VKS cluster] --> B[pvCSI, no vCenter credentials]
  B --> C[Mirror PVC in the vSphere Namespace]
  C --> D[CNS CSI on the Supervisor]
  D --> E[CNS in vCenter]
  E --> F[Datastore matched by storage policy]
  F --> G[PV shows Bound in cluster and namespace]
One kubectl apply crosses four components before a disk exists. Every one of them can be the thing that failed.

Now the failure, because a happy path teaches nothing. Here is the war story that made me build the lookup table further down. A customer of mine had a PostgreSQL StatefulSet stuck Pending for three days. Two engineers had rewritten the Helm values twice and opened a support case against the chart. I asked for one command.

$ kubectl get pvc -n data NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-postgres-0 Pending vks-high-performance 3d $ kubectl describe pvc data-postgres-0 -n data | tail -4 Events: Type Reason Age From Message —- —— —- —- ——- Warning ProvisioningFailed 12s (x842 over 3d) persistentvolume-controller storageclass.storage.k8s.io "vks-high-performance" not found

Not a chart problem. Not a Kubernetes problem. Somebody had assigned only the default storage policy to that vSphere Namespace, so the high performance class was never generated and never propagated. Fix time in the vSphere Client was under a minute: open the namespace, add the second storage policy, and watch the class appear in the cluster without touching the StatefulSet. Three days of engineering time against forty seconds of vCenter work, and the whole gap was caused by nobody in the room knowing which object creates a storage class.

# What green looks like, verified from inside the cluster $ kubectl get storageclass vks-high-performance NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION vks-high-performance csi.vsphere.vmware.com Delete Immediate true $ kubectl get pvc -n data NAME STATUS VOLUME CAPACITY STORAGECLASS data-postgres-0 Bound pvc-8f2c1a44-6d17-4a0e-9f7c-5b21c0a97e10 20Gi vks-high-performance # And confirmed from the Supervisor side, where the mirror claim lives $ kubectl get pvc -n vks-prod | grep postgres vks-prod-01-data-postgres-0 Bound pvc-8f2c1a44-6d17-4a0e-9f7c-5b21c0a97e10 20Gi

Backing that out is undramatic and worth rehearsing. Removing a storage policy from a namespace removes the class from the cluster, but it does not delete volumes already bound through it, and a pod that later tries to reschedule will fail in a way that looks like a scheduling fault. Drain the workload first, then remove the policy, then confirm no PVC still references the class.

Helm charts, packages and a service mesh footnote

Helm is a package manager for Kubernetes and it works on a VKS cluster exactly as it works anywhere else, which is precisely why it earns a mention in the objective and almost no special handling. You are cluster admin inside a VKS cluster, so helm install behaves normally. Two things are different and both bite.

First, Pod Security Admission is enabled by default on VKS clusters, with the enforcement level set through the podSecurityStandard variable on the cluster, and system namespaces such as kube-system and tkg-system excluded. Any community chart that assumes it can run privileged will be refused, and the refusal appears on the ReplicaSet rather than on the helm command, which is why people misread it.

$ helm install metrics oci://registry.example.local/charts/legacy-exporter -n obs NAME: metrics STATUS: deployed $ kubectl get pods -n obs No resources found in obs namespace. $ kubectl describe rs -n obs metrics-legacy-exporter | tail -3 Warning FailedCreate 8s replicaset-controller Error creating: pods "metrics-legacy-exporter-2fj9x" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "exporter" must set securityContext.allowPrivilegeEscalation=false), runAsNonRoot != true

Helm reported success because Helm did its job. It created a Deployment. Admission control killed the pods a layer below. Fix it by setting securityContext fields in the chart values, or by relaxing enforcement for that one namespace with a label, and choose the first option unless you enjoy explaining the second to an auditor.

Second, Helm is not the only packaging system in play. VKS ships a set of standard packages delivered through Carvel style repositories and driven by the VCF CLI, which is a separate mechanism with its own versioning and its own default values files. Istio, the service mesh, arrives that way rather than as a plain Helm chart.

# Discovery only. Installing a mesh properly is objective 3.6, later in this series. $ vcf package available get istio.kubernetes.vmware.com -n tkg-system NAME VERSION RELEASED-AT istio.kubernetes.vmware.com 1.25.3+vmware.1-vks.1 2025-05-20 $ vcf package available get istio.kubernetes.vmware.com/1.25.3+vmware.1-vks.1 –default-values-file-output istio-data-values.yaml -n tkg-system

For Objective 1.2 you need to know what a service mesh does and that VKS offers Istio as a supported package, not how to run one. A mesh puts a proxy in the request path so that traffic between services can be encrypted, retried, split and observed without application code changing. Istio gives you two shapes of data plane: the sidecar model, where a proxy container joins every workload pod, and the ambient model, where a shared node level proxy handles traffic instead. Sidecars cost memory per pod and ambient costs you a newer, less familiar failure surface. Part 15 of this series takes that decision apart properly against Objective 3.6, and package repository mechanics belong to Objective 4.8.

Failure to cause lookup for objective 1.2

This is the artifact to keep. Every row maps a Kubernetes symptom to the vSphere object that actually caused it, which is the exact translation the objective is testing and the exact translation Section 5 will demand of you under time pressure.

Symptom in the clusterObject that caused itRemediation
storageclass.storage.k8s.io not foundStorage policy never assigned to the vSphere NamespaceAdd the policy to the namespace, class appears within seconds
PVC Bound but pod stays Pending on a zoned clusterImmediate binding class placed the disk in the wrong vSphere ZoneRecreate the claim against the WaitForFirstConsumer edition
Service EXTERNAL-IP stuck pendingLoad balancer VIP range exhausted at Supervisor levelReclaim VIPs or resize the pool, nothing to fix in the cluster
Pods created by Helm never appearPod Security Admission enforcement on the clusterSet securityContext in chart values, read the ReplicaSet events
Node NotReady with no useful cluster eventsMachine or VirtualMachine object failing on the SupervisorSwitch context, describe the Machine, then read vSphere tasks
Topology spread constraints silently ignoredvSphere Zones not mapped, so no zone labels on nodesVerify zone labels with kubectl get nodes before blaming the scheduler

Exam focus for objective 1.2

Objective 1.2, Use Kubernetes architecture, networking, storage, service mesh, and Helm:
What it expects you to be able to do. Look at a Kubernetes object or a piece of CLI output and correctly attribute it to the layer that produced it. CNI, load balancer, CSI path, admission control, packaging system.
Item types it suits. Matching and drag and drop, because the material is naturally paired: endpoint to provider, symptom to layer, component to where it runs. Point and click on a diagram of the storage path is a natural fit too.
Trap that catches experienced admins. Answering from generic Kubernetes knowledge. An upstream Kubernetes administrator knows a storage class is created by an administrator applying YAML, so given a missing storage class they reach for kubectl apply. On VKS that answer is wrong every time, because the Supervisor generates the class from a vSphere Namespace policy assignment and will reconcile your handmade one away.
Cheap marks. Antrea is the default CNI and Calico is the only alternative. VKS ships no ingress controller. pvCSI holds no vCenter credentials. Those four facts are small, testable and frequently missed.

Objective checkpoint

Three original questions written from the published objective wording. Cover the reasoning, answer, then check.

1. A VKS cluster is provisioned with no cni field in its manifest. Which container network interface is running, and what would it take to change it?
Antrea, because it is the system default and needs no declaration. Changing it means specifying Calico in the cluster YAML at provisioning time or altering the Supervisor wide default. Calico is the only supported alternative.

2. A developer reports that a PVC in their VKS cluster has been Pending for an hour with a message that the storage class was not found. Which object should the administrator inspect first?
The vSphere Namespace on the Supervisor. Storage classes inside a VKS cluster are generated from storage policies assigned to that namespace, so a missing class is a missing policy assignment, not a cluster fault.

3. Which component in the persistent volume path holds no vCenter credentials, and why does that matter operationally?
pvCSI, which runs inside the VKS cluster and is configured with a namespace service account. It relays requests to CNS CSI on the Supervisor. It matters because a tenant with cluster admin still cannot reach vCenter storage directly, so the blast radius of a compromised cluster stops at the namespace.

Learn the inheritance chain, not the Kubernetes glossary

My recommendation for this objective is blunt, and it contradicts most study advice I see posted. Do not spend a fortnight on a general Kubernetes course before sitting 3V0-24.25. If you already know what a Deployment, a Service and a PVC are, you know enough Kubernetes for this exam. Spend that fortnight instead building the mental map of which vSphere object produces which Kubernetes object, because that map is what the other 32 objectives silently assume and it is not taught in any Kubernetes course.

A clean result for this part looks like four things. You can switch between the Supervisor context and the cluster context without hesitating. You can name the provider for every row in the endpoint table from memory. You can point at any storage class in your cluster and name the vSphere storage policy behind it. And you can explain, out loud, why a claim in your cluster becomes a disk in vCenter without your cluster ever holding a vCenter credential.

Tonight, in your own lab, run kubectl get storageclass in a VKS cluster and then open the matching vSphere Namespace in the vSphere Client. Count the classes, count the policies, and confirm the arithmetic holds. If it does not, you have found something worth understanding before Part 4 turns to reference architecture. Next part we leave the terminal and design the estate: NSX, VDS and zones against Objective 1.3.

VCAP-VKS Exam Series · Part 3 of 34
« Previous: Part 2  |  Guide  |  Next: Part 4 »

References

VKS Cluster Networking, Broadcom TechDocs, VCF 9.0
Storage for VKS Clusters, Broadcom TechDocs, VCF 9.0
Install Istio on VKS Clusters, Broadcom TechDocs
VMware Cloud Foundation VKS Administrator Exam Guide, 3V0-24.25

About The Author


Discover more from Journal of Intelligent Infrastructure

Subscribe to get the latest posts sent to your email.

Leave a Reply

Your email address will not be published. Required fields are marked *

Architect’s Toolkit

About the Author

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

Discover more from Journal of Intelligent Infrastructure

Subscribe now to keep reading and get access to the full archive.

Continue reading