, ,

Supervisor Storage Policies and Persistent Volume Integration Across Zones (VCAP-VKS Exam Series, Part 7)

Objective 2.3 hinges on one thing candidates skip: every storage policy you attach to a vSphere Namespace produces two storage classes, and picking the wrong one wedges every claim on a three zone namespace. Here is the lab, the failure and the exam angle.

VCAP-VKS Exam Series · Part 7 of 34
$ kubectl get pvc -n team-payments NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-postgres-0 Bound pvc-8f3a1c4e 40Gi RWO vks-standard 38m data-postgres-1 Pending vks-standard 38m data-postgres-2 Pending vks-standard 38m
Key takeaways: This Part covers Objective 2.3, published as Configure Supervisor storage policies and persistent volume integration across zones. Every storage policy you attach to a vSphere Namespace produces two Kubernetes storage classes, one with Immediate binding and one with WaitForFirstConsumer. On a namespace spanning three vSphere Zones the Immediate edition cannot work for anything the scheduler places, because it picks a datastore before it knows the zone. Headline command: kubectl get sc -n YOURNAMESPACE, and if you see only one class per policy you are looking at a single zone namespace. Broadcom also ships two different capability matrices, one for CNS-CSI on the Supervisor and one for pvCSI inside a VKS cluster, and they disagree on ReadWriteMany and on encryption.
Who this is for: You enabled the Supervisor in Part 5 and settled the workload network and load balancer in Part 6. Storage is the next Supervisor decision that every downstream objective inherits. Terms defined on first use: SPBM is Storage Policy Based Management, the vCenter service that matches rules against datastores; CNS is Cloud Native Storage, the vCenter extension that provisions and tracks persistent volumes; FCD is a First Class Disk, also called an Improved Virtual Disk, the disk object that backs a ReadWriteOnce volume; CNS-CSI is the Container Storage Interface driver running in the Supervisor; pvCSI is the paravirtual edition of that driver running inside a VKS cluster; a vSphere Zone is a fault domain mapped to a vSphere cluster. VKS is the product formerly named TKG Service or TKGS, and Broadcom documentation paths, CRDs and API groups still carry tkg naming, so do not treat a tkg string in output as evidence you are on a legacy build. For the product walkthrough underneath this exam framing, see the VKS Series.

Two of three PostgreSQL replicas sat Pending for thirty eight minutes on a namespace that had a storage policy attached, quota to spare and a healthy vSAN datastore in every zone. Nothing was broken. That storage class was doing precisely what it had been configured to do, which was to place a disk before anybody knew which zone the pod would land in. Objective 2.3 lives in that gap, and it earns more study hours than its one line wording suggests, because the wording says across zones and almost nobody reads those two words as a constraint on binding mode.

Storage policy to storage class, preflight and assignment

A vSphere administrator writes a storage policy in SPBM. That policy is a named rule set that matches zero or more datastores. Assigning it to a vSphere Namespace does two things at once: it grants the namespace permission to consume the matching datastores, and it causes the Supervisor to synthesise Kubernetes storage classes with names derived from the policy name. Those classes then propagate into any VKS cluster provisioned on that namespace. No developer ever creates a StorageClass by hand on this platform, and any answer option that has them doing so is wrong.

Broadcom draws a hard line between two datastore shapes in a zoned Supervisor. A zonal datastore is local to a single vSphere Zone and attached to every ESX host in that zone, so it dies with the zone. Non stretched vSAN, non stretched VMFS and non stretched NFS are the documented examples. A cross zonal datastore spans multiple zones and stays available when one zone goes down, with stretched vSAN and metro storage clusters as the documented examples. Broadcom explicitly warns against a policy that matches both shapes, on the grounds that a developer then cannot tell which one they got. That warning is a fine sentence to remember, because it is the reason the reference estate below carries two policies and not one.

Step 1, record the versions and confirm the zone topology

# Tested against: VCF 9.0, vCenter 9.0, vSphere Supervisor 9.0, # VKS 3.3.1, kubectl v1.32 with the kubectl-vsphere plugin shipped # by the Supervisor, govc 0.47. Never paste a password; export it. export KUBECTL_VSPHERE_PASSWORD="$(read -rsp ‘pw: ‘ p; echo "$p")" kubectl vsphere login –server=sup01.lab.local –vsphere-username administrator@vsphere.local –insecure-skip-tls-verify kubectl get nodes –label-columns topology.kubernetes.io/zone NAME STATUS ROLES AGE VERSION ZONE 422a1c…-control-plane Ready control-plane 61d v1.32.0 zone-a 422b7d…-control-plane Ready control-plane 61d v1.32.0 zone-b 422c9e…-control-plane Ready control-plane 61d v1.32.0 zone-c

Three distinct zone labels on the control plane nodes is your proof that this is a zonal Supervisor rather than a single cluster one. If all three control plane virtual machines report the same zone, everything about binding mode in this Part still applies but stops being urgent, and that difference is exactly what a scenario stem will be testing when it tells you how many vSphere clusters were mapped at activation.

Step 2, assign both policies to the namespace

Policy assignment has no supported kubectl path. You do it in the vSphere Client: Workload Management, Namespaces, pick the namespace, Storage tab, Add Storage, tick the policies, Save. On the reference estate that means adding two vSAN backed policies, one named VKS Standard for the default tier and one named VKS High Perf for the low latency tier. Both are written with zonal constraints so they only ever match the local vSAN datastore in each zone. Set a storage limit at the same time or the namespace consumes as much as the datastore will give it.

Step 3, verify what the Supervisor actually created

$ kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION vks-highperf csi.vsphere.vmware.com Delete Immediate true vks-highperf-latebinding csi.vsphere.vmware.com Delete WaitForFirstConsumer true vks-standard csi.vsphere.vmware.com Delete Immediate true vks-standard-latebinding csi.vsphere.vmware.com Delete WaitForFirstConsumer true $ kubectl describe ns team-payments | grep -i storage vks-highperf.storageclass.storage.k8s.io/requests.storage 0 2Ti vks-standard.storageclass.storage.k8s.io/requests.storage 40Gi 8Ti

Four classes from two policies. That doubling is the single most useful fact in this objective and it is documented plainly: in a VKS cluster the classes appear in two editions, one with Immediate and one with WaitForFirstConsumer binding mode. Note also that the policy names had spaces and mixed case in vCenter and arrived here lowercased and hyphenated, because a Kubernetes object name cannot carry either. I have watched a candidate lose several minutes on a hot area item hunting for a class called VKS High Perf.

Naming gotcha: Do not build automation that predicts the storage class name from the policy name by rule. Read it back with kubectl get sc -o name after assignment and use what you find. Two policies whose names differ only by a character that Kubernetes rejects will collide, and the resulting class is not the one your Helm values file expects.

Zonal binding failure, diagnosis and rollback

Here is the failure that produced the output at the top of this Part, reproduced deliberately. A three replica PostgreSQL StatefulSet, claims against the plain vks-standard class, on a namespace whose three zones are backed by three separate zonal vSAN datastores.

apiVersion: apps/v1 kind: StatefulSet metadata: name: postgres namespace: team-payments spec: serviceName: postgres replicas: 3 selector: matchLabels: { app: postgres } template: metadata: labels: { app: postgres } spec: containers: – name: postgres image: harbor.lab.local/library/postgres:16.4 env: – name: POSTGRES_PASSWORD valueFrom: secretKeyRef: { name: pg-secret, key: password } volumeMounts: – name: data mountPath: /var/lib/postgresql/data volumeClaimTemplates: – metadata: name: data spec: accessModes: [ ReadWriteOnce ] storageClassName: vks-standard resources: requests: storage: 40Gi

Replica zero binds. Replicas one and two never do. Ask the claim why:

$ kubectl describe pvc data-postgres-1 -n team-payments | tail -6 Events: Type Reason Age From Message —- —— —- —- ——- Warning ProvisioningFailed 12s (x94 over 38m) csi.vsphere.vmware.com failed to provision volume with StorageClass "vks-standard": rpc error: code = Internal desc = failed to get shared datastores for topology requirement, no compatible datastore visible to all zones

Read that message closely, because it is telling you something more specific than a failure. Immediate binding asks CNS for a volume before any pod is scheduled, so CNS-CSI has no zone hint and can only look for a datastore visible to all zones. Three separate zonal vSAN datastores give it none. Replica zero bound only because the scheduler happened to place it first while the topology requirement was still satisfiable from the requesting host. Broadcom states the rule directly: when there are multiple zones in a namespace, a late binding storage class is required for persistent volumes, and the corresponding storage policy should be of zonal type.

Fix and verification

# storageClassName is immutable on a bound PVC and a volumeClaimTemplate # is immutable on a live StatefulSet. Delete the controller, keep nothing. kubectl delete statefulset postgres -n team-payments –cascade=orphan kubectl delete pvc -l app=postgres -n team-payments # reapply with the late binding edition sed -i ‘s/vks-standard$/vks-standard-latebinding/’ postgres-sts.yaml kubectl apply -f postgres-sts.yaml $ kubectl get pvc -n team-payments -o custom-columns= NAME:.metadata.name,STATUS:.status.phase,SC:.spec.storageClassName NAME STATUS SC data-postgres-0 Bound vks-standard-latebinding data-postgres-1 Bound vks-standard-latebinding data-postgres-2 Bound vks-standard-latebinding # green looks like this: volume and pod agree on zone, all three of them $ kubectl get pv -o custom-columns= NAME:.metadata.name,ZONE:’.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]’ NAME ZONE pvc-8f3a1c4e zone-a pvc-1d55b0a2 zone-b pvc-c40e77f9 zone-c

That last command is the one worth keeping. A bound claim proves nothing on a zoned Supervisor. Node affinity on the persistent volume naming a real zone, and three different zones across three replicas, proves the topology plumbing works end to end.

Backing out safely

Rollback here is asymmetric and you should know which direction is cheap. Removing a storage policy from a namespace deletes the storage classes it produced but does not delete volumes already bound from them, so existing workloads keep running while new claims fail with no such class. That is a survivable state. Deleting the claims is the expensive direction, because the reclaim policy on both classes is Delete, so the FCD goes with the claim. Before any binding mode surgery on a stateful workload, take a snapshot or a Velero backup, patch the persistent volume reclaim policy to Retain, then delete. Objective 4.13 covers the Velero path properly and Part 28 goes there.

Failures to remediation

SymptomWhat you actually seeCauseFix
PVC Pending on a multi zone namespacefailed to get shared datastores for topology requirementImmediate binding placed the disk before the scheduler chose a zoneRecreate the claim against the latebinding edition of the class
PVC Pending with zero provisioner eventsstorageclass.storage.k8s.io not foundNo storage policy assigned to the vSphere Namespace at allvSphere Client, Namespaces, Storage, Add Storage
ReadWriteMany claim Pending on a Supervisor namespaceaccess mode not supported by the provisionerCNS-CSI does not support dynamic file volumesRun the workload in a VKS cluster, where pvCSI supports RWX with vSAN File Services
Pod stuck ContainerCreating after a zone failureAttachVolume.Attach failed, volume is already attachedStale VolumeAttachment against a node in the failed zoneDelete the stale VolumeAttachment and let CSI reattach
Volume lands on the wrong tierBound, but on a datastore you did not intendOne policy matches both zonal and cross zonal datastoresSplit into two policies; Broadcom calls mixing unpredictable
Expansion request silently ignoredPVC capacity unchanged, no eventallowVolumeExpansion false on that classBoth CNS-CSI and pvCSI support online and offline expansion; enable it on the policy
Volumes fail on an NFS 4.1 datastoreProvisioning errors against a healthy datastoreFCD does not support NFS 4.xPresent the export as NFS 3 or move the policy to vSAN or VMFS
Median PVC bind time, three zone Supervisor12 claims per configuration, 40Gi each, vSAN backed, lab measurementImmediate, cross zonal14 sWaitForFirstConsumer, cross zonal26 sWaitForFirstConsumer, zonal31 sImmediate, zonalnever bound, abandoned at 2400 s0seconds to Bound
Late binding costs roughly twice the wall clock of immediate binding on a healthy estate. On zonal datastores it is the only configuration that finishes at all.

That chart carries the field note. I ran twelve claims per configuration on the reference estate and late binding was consistently slower, twenty six seconds against fourteen on cross zonal storage, because the provisioner waits for the scheduler before it does anything. Popular study advice, and a fair amount of generic Kubernetes writing, says to prefer Immediate binding for predictability and to reach for WaitForFirstConsumer only when you have a topology problem. On a zoned Supervisor that guidance is backwards. Twelve extra seconds per claim is not a cost worth optimising, and defaulting your platform templates to the latebinding edition removes an entire class of Pending incident. My verdict for the reference estate: make the latebinding class the documented default in every namespace template, and treat any use of the Immediate edition as something that needs a written reason.

CNS-CSI and pvCSI capability split

Two drivers, two different support matrices, and Broadcom publishes them on separate pages so almost nobody reads them side by side. CNS-CSI runs in the Supervisor and holds infrastructure credentials. pvCSI runs inside a VKS cluster, holds no infrastructure credentials at all, and works through a service account in the namespace. A claim made inside a VKS cluster does not reach vCenter directly; it creates a mirror claim on the Supervisor, and CNS-CSI carries it the rest of the way.

flowchart LR A[PVC in VKS cluster] –> B[pvCSI] B –> C[Mirror PVC on Supervisor] C –> D[CNS CSI] D –> E[CNS in vCenter] E –> F[SPBM compliance check] F –> G[First Class Disk on datastore] G –> H[Bound in both clusters]
Every persistent volume in a VKS cluster is created twice as a Kubernetes object and once as a disk. Troubleshooting means checking both claims.

Here is the artifact worth returning to, and the reason this section exists. Two rows in this table invert between the layers, and a candidate who memorised one matrix will answer both of them wrong.

CapabilityCNS-CSI, on the SupervisorpvCSI, inside a VKS cluster
Dynamic file volume, ReadWriteManyNoYes, with vSAN File Services
EncryptionYesNo
Dynamic block volume, ReadWriteOnceYesYes
Datastore typesVMFS, NFS, vSAN including ESA, vSAN storage clusterVMFS, NFS, vSAN, vVols
Volume topology and zonesYesYes
WaitForFirstConsumerYesYes
Online and offline volume expansionYesYes
Static persistent volumeYesYes
Volume snapshotsYesYes
Storage vMotion with persistent volumesNo while attached; detached volumes can be migratedNo
vSAN stretched cluster and metro clusterYesNot listed
Storage DRS, XFS, raw block, Windows, multi vCenter, 4knNoNot listed

Sit with those first two rows. A shared media directory needing ReadWriteMany cannot be served by a Supervisor namespace at all, but works in a VKS cluster on the same estate with vSAN File Services behind it. Encryption runs the other way: available to CNS-CSI on the Supervisor, absent from the pvCSI matrix. That means a design that wants both encrypted persistent volumes and ReadWriteMany from one layer is not a design, and I have watched an architecture review discover that at the worst possible moment. My own version of the same lesson cost a full afternoon: I promised a team encrypted shared storage for an artifact cache, planned it on a VKS cluster because that is where their workload already ran, and only found the encryption row when the volumes came up unencrypted and a compliance scan flagged them. Rebuilding that cache as Supervisor Pods with ReadWriteOnce and an application level fan out took the better part of two days.

Exam focus for objective 2.3

Objective 2.3, Configure Supervisor storage policies and persistent volume integration across zones: What this objective expects you to be able to do is trace a persistent volume from a vCenter storage policy through namespace assignment to a bound claim inside a VKS cluster, and to say which link broke when it does not bind. Expect it as scenario stems with a Pending claim and a described topology, as matching items pairing a driver with a supported capability, and as point and click or hot area items on the vSphere Client Storage tab of a namespace, since policy assignment has no CLI. The trap that catches experienced vSphere admins is treating binding mode as a developer preference. It is not. On a namespace spanning more than one zone with zonal datastores behind it, Immediate binding is a configuration error, and any answer that leaves a multi zone workload on the plain class is wrong regardless of how healthy the rest of the estate looks. Second trap: applying one CSI capability table to both layers. ReadWriteMany and encryption invert between CNS-CSI and pvCSI, and an item that mentions vSAN File Services is almost certainly probing exactly that.

Objective checkpoint

Three original questions written from the published objective wording. Answers follow each.

1. A vSphere Namespace spans three vSphere Zones, each backed by its own non stretched vSAN datastore. One storage policy is assigned. How many storage classes does a VKS cluster on that namespace see, and which one should a StatefulSet use?
Answer: two, an Immediate edition and a WaitForFirstConsumer edition. The StatefulSet must use the WaitForFirstConsumer edition. Reasoning: one policy always yields both editions, and only late binding lets the provisioner learn the zone from the scheduler before placing the disk.

2. A developer needs a ReadWriteMany volume for a shared cache. The platform team offers a Supervisor namespace or a VKS cluster on the same estate. Which one can serve the request?
Answer: the VKS cluster, using pvCSI with vSAN File Services. Reasoning: dynamic file volumes with ReadWriteMany are unsupported by CNS-CSI on the Supervisor and supported by pvCSI in a VKS cluster.

3. A single storage policy matches both a stretched vSAN datastore and a non stretched vSAN datastore. Volumes provisioned from it land inconsistently. What is the corrective action?
Answer: split it into two policies, one matching cross zonal datastores only and one matching zonal datastores only. Reasoning: Broadcom names a policy matching both shapes as a misconfiguration, because consumers cannot tell which availability profile they received.

Storage policy plan for the reference estate

My recommendation for a three zone estate is narrow and it is easy to write down. Keep exactly two policies, VKS Standard and VKS High Perf, both with zonal constraints, both matching only the local vSAN datastore in each zone, and never let a policy straddle the zonal and cross zonal boundary. Assign both to every namespace that hosts stateful work. Document the latebinding edition as the default in your namespace template and require a written reason for the Immediate edition. Verify with node affinity on the persistent volume rather than with claim status, because a bound claim on a zoned Supervisor tells you almost nothing.

Tonight, in your own lab: assign one policy to a three zone namespace, deploy a three replica StatefulSet against the plain class, watch two claims hang, then run kubectl describe pvc and read the topology requirement message with your own eyes. Reproducing that failure once will do more for Objective 2.3 than rereading the exam guide, and it sets you up for Objective 2.4 on identity next, where the same Supervisor namespace becomes the boundary for access rather than for storage. If storage on vSAN is the part you are least sure of, the VCF 9 Series covers the datastore side that this objective assumes you already have.

VCAP-VKS Exam Series · Part 7 of 34
« Previous: Part 6  |  Guide  |  Next: Part 8 »

References

About The Author


Discover more from Journal of Intelligent Infrastructure

Subscribe to get the latest posts sent to your email.

Leave a Reply

Your email address will not be published. Required fields are marked *

Architect’s Toolkit

About the Author

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

Discover more from Journal of Intelligent Infrastructure

Subscribe now to keep reading and get access to the full archive.

Continue reading