, ,

VKS Storage Strategies with Dynamic and Static Persistent Volumes (VCAP-VKS Exam Series, Part 26)

Objective 4.11 in a runnable lab: build a dynamic claim, adopt a retained volume statically through a Supervisor PVC handle, and expand a live PostgreSQL volume without losing the pod. Includes the failure lookup and the volumeHandle trap that catches vSphere CSI veterans.

VCAP-VKS Exam Series · Part 26 of 34

A claim in my lab reported 40Gi last Tuesday while df inside the pod consuming it reported 20G. Neither number was wrong. Kubernetes had finished half of a volume expansion and was waiting on me to finish the other half, and nothing in the default output said so unless I went looking for the conditions. That gap, between what the control plane has already done and what the node has not yet done, is a large share of what objective 4.11 is actually testing.

Who this is for: A candidate who took volume snapshots in Part 24 and moved the Supervisor underneath the estate in Part 25, and now has to reason about the volume layer itself. This Part covers Objective 4.11, published wording Create storage strategies using dynamic and static persistent volumes with support for expansion. Terms defined on first use: VKS is vSphere Kubernetes Service, the product formerly named TKG Service or TKGS, and on VCF 9.0 that rename is still unfinished, so tkg survives in namespaces, CRD groups and documentation URLs; a PV is a PersistentVolume, the cluster object representing a piece of storage; a PVC is a PersistentVolumeClaim, a request for one; a StorageClass names the provisioner and parameters used to satisfy a claim; CNS is Cloud Native Storage, the vSphere subsystem that tracks container volumes; CSI is the Container Storage Interface; pvCSI is the paravirtual CSI driver a VKS cluster runs so its storage calls are proxied up to the Supervisor rather than talking to vCenter directly; FCD is a First Class Disk, a vSphere virtual disk with an independent lifecycle; RWO is ReadWriteOnce and RWX is ReadWriteMany; a reclaim policy decides what happens to backing storage when a claim is deleted.
Key takeaways: Objective 4.11 uses the plural word strategies on purpose, so the testable skill is choosing between dynamic and static provisioning and defending the choice, not typing one manifest. A StorageClass inside a VKS cluster is not something you author; it is a mirror of a vSphere storage policy assigned to the vSphere Namespace on the Supervisor, and every Pending claim traces back to a break in that chain. Static provisioning in VKS is not the vanilla vSphere CSI pattern: spec.csi.volumeHandle takes the name of a PVC on the Supervisor, in the vSphere Namespace hosting the cluster, and that PVC must not be attached to anything. Expansion runs in two stages, controller then node, and stage two frequently needs a pod restart before df changes. Supervisor does not expand ReadWriteMany volumes and does not expand in tree or migrated volumes at all. Headline command: kubectl patch pvc pgdata-high –namespace databases –type merge -p with a new spec.resources.requests.storage value.

Storage vocabulary this objective leans on

Every storage failure I have watched a candidate fumble in a lab came from treating a VKS StorageClass as a Kubernetes object they own. It is not. On VCF 9.0 a storage class inside a workload cluster is the far end of a four hop chain that starts in vCenter, and any hop can be intact while the next one is missing. You create a vSAN storage policy. You assign that policy to a vSphere Namespace. Assigning it causes a matching StorageClass to appear on the Supervisor. Listing that class in the VKS cluster specification causes it to appear inside the workload cluster. Only then does a claim bind.

This series traces every workload cluster symptom back to the Supervisor object that caused it, and storage is where that discipline pays best. A claim stuck Pending in a workload cluster almost never has a workload cluster cause. Part 7 of this series walked the policy side of that chain in detail, so I will not repeat it here beyond the link: Supervisor storage policies and persistent volume integration across zones. If CNS itself is unfamiliar, the product walkthrough underneath this study series covers it: VKS complete guide.

Two facts about the chain matter for the exam and get skipped in tutorials. First, the mirrored class inside the workload cluster is reconciled, so editing it by hand is pointless; your edit is reverted and you lose ten minutes convincing yourself the cluster is broken. Second, allowVolumeExpansion is a property of the class, and a claim created while that property was false is not retroactively resizable when you later flip it. Sequence matters more than state.

flowchart TD
  A[vSAN storage policy in vCenter] --> B[Policy assigned to vSphere Namespace]
  B --> C[StorageClass appears on Supervisor]
  C --> D[Class listed in VKS Cluster spec]
  D --> E[StorageClass visible in workload cluster]
  E --> F{PVC created}
  F -->|class present and quota free| G[Bound, pvCSI proxies to Supervisor]
  F -->|any hop missing| H[Pending forever, no provisioner events]

Four hops from a vCenter policy to a bound claim. Diagnose downward from the top, never upward from the claim.

Preflight checks before any claim is created

Where the lab stands: Part 25 left the Supervisor updated and the three zone estate healthy, with the pg-orders PostgreSQL StatefulSet running on a dynamic claim backed by the default vSAN policy. This Part adds the high performance policy as a second consumable class, adopts a volume that was deliberately left behind, and grows a live database volume. Everything below assumes the versions in the first block.

# Versions this Part was tested against VCF 9.0 vCenter 9.0.0.0100 Supervisor v1.32.9 VKS (TKG Service) 3.3.1 VKr node image v1.32.5—vmware.1-fips.1 kubectl v1.32.4 kubectl-vsphere 9.0.0 # Preflight 1: what the Supervisor is willing to offer this namespace kubectl config use-context 10.60.12.20 kubectl get storageclass NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE vsan-default csi.vsphere.vmware.com Delete WaitForFirstConsumer true 61d vsan-high-perf csi.vsphere.vmware.com Retain WaitForFirstConsumer true 4h12m

Reading that output is half the preflight. Two classes exist on the Supervisor because two policies are assigned to the vSphere Namespace. Both allow expansion. Note that vsan-high-perf carries Retain rather than Delete, which is a deliberate choice I will defend later and which most tutorials tell you not to make. Note also the binding mode: WaitForFirstConsumer means a claim will sit Pending until a pod schedules, which on a zonal Supervisor is correct behaviour and not a fault. Candidates report that as broken every single time.

Now the failure worth practising deliberately, because it is the single most common storage symptom on this platform. Assign a new policy in vCenter, forget to add it to the workload cluster specification, and the class exists one hop above where your claim is looking.

# Preflight 2: same question, asked inside the workload cluster kubectl config use-context vks-prod-01 kubectl get storageclass NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE vsan-default (default) csi.vsphere.vmware.com Delete WaitForFirstConsumer true 38d # The high performance class is on the Supervisor but not here. Prove the consequence. kubectl apply –filename pgdata-high.yaml –namespace databases kubectl describe pvc pgdata-high –namespace databases | tail -6 Events: Type Reason Age From Message —- —— —- —- ——- Warning ProvisioningFailed 9s (x3 over 21s) persistentvolume-controller storageclass.storage.k8s.io "vsan-high-perf" not found # Fix at the Supervisor, not here: add the class to the cluster spec kubectl config use-context 10.60.12.20 kubectl patch cluster vks-prod-01 –namespace ns-platform –type merge -p ‘{"spec":{"topology":{"variables":[{"name":"storageClasses","value":["vsan-default","vsan-high-perf"]}]}}}’

Watch the error text carefully. It comes from persistentvolume-controller, not from a CSI driver, because no driver was ever consulted. When an exam scenario hands you a Pending claim and an events block with no CSI messages in it, the answer is upstream of the driver every time.

Numbered run, dynamic claim, static adoption, live expansion

Step 1. Dynamic claim on the high performance class

Dynamic provisioning is the default answer and should be, because it is the only path where the platform owns the whole lifecycle. Create the claim, let a pod schedule, watch it bind.

# pgdata-high.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: pgdata-high namespace: databases spec: accessModes: – ReadWriteOnce storageClassName: vsan-high-perf resources: requests: storage: 20Gi kubectl apply –filename pgdata-high.yaml kubectl get pvc pgdata-high –namespace databases NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE pgdata-high Bound pvc-7a41c9e2-5db0-4b17-9c3a-1e0f4a77b2d1 20Gi RWO vsan-high-perf 47s

Step 2. Static adoption of a volume that already exists

Static provisioning is where veterans lose marks, so slow down here. In vanilla vSphere CSI you point a PersistentVolume at an FCD identifier. In a VKS cluster you do not, because the workload cluster runs pvCSI and has no direct view of vCenter. Broadcom documentation is explicit: spec.csi.volumeHandle must carry the name of a PersistentVolumeClaim created on the vSphere Namespace where the target cluster lives, and that claim must not be attached to any pod. Same field name, completely different value.

In our lab that Supervisor claim is called archive-vol-01. It was left behind on purpose when a previous cluster was deleted, which the Retain policy on vsan-high-perf made possible. Four fields have to agree or the bind silently never happens: capacity, access mode, storageClassName, and the claimRef pointing at the exact namespace and name of the claim you are about to create.

# Confirm the backing claim exists on the Supervisor and is unattached kubectl config use-context 10.60.12.20 kubectl get pvc archive-vol-01 –namespace ns-platform NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE archive-vol-01 Bound pvc-0d16185f-626b-4788-a55e-4d3e48245226 50Gi RWO vsan-high-perf 19d # static-archive.yaml, applied to the WORKLOAD cluster apiVersion: v1 kind: PersistentVolume metadata: name: static-archive-pv annotations: pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com spec: storageClassName: vsan-high-perf capacity: storage: 50Gi accessModes: – ReadWriteOnce persistentVolumeReclaimPolicy: Retain claimRef: namespace: databases name: static-archive-pvc csi: driver: "csi.vsphere.vmware.com" volumeAttributes: type: "vSphere CNS Block Volume" volumeHandle: "archive-vol-01" nodeAffinity: required: nodeSelectorTerms: – matchExpressions: – key: topology.kubernetes.io/zone operator: In values: – zone-a — apiVersion: v1 kind: PersistentVolumeClaim metadata: name: static-archive-pvc namespace: databases spec: accessModes: – ReadWriteOnce storageClassName: vsan-high-perf resources: requests: storage: 50Gi volumeName: static-archive-pv

That nodeAffinity block is not optional on our estate and is easy to leave out. Three vSphere Zones back this Supervisor, the backing volume physically lives in zone-a, and without the affinity term a scheduler will happily place the consuming pod on a node in zone-c where the disk cannot be attached. You get a pod wedged in ContainerCreating and an attach error that reads like a driver bug rather than a placement mistake.

Step 3. Expand a live volume and hit the wall on purpose

Expansion is one field. Finishing expansion is a different problem. Patch the claim, then immediately go looking for the condition rather than for a success message, because there is no success message.

# First, the failure worth seeing: a class that forbids resize kubectl patch pvc legacy-logs –namespace databases –type merge -p ‘{"spec":{"resources":{"requests":{"storage":"40Gi"}}}}’ Error from server (Forbidden): persistentvolumeclaims "legacy-logs" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize # Now the real one, on a class with allowVolumeExpansion true kubectl patch pvc pgdata-high –namespace databases –type merge -p ‘{"spec":{"resources":{"requests":{"storage":"40Gi"}}}}’ persistentvolumeclaim/pgdata-high patched kubectl get pvc pgdata-high –namespace databases -o jsonpath='{.status.capacity.storage}{" "}{.status.conditions[*].type}{"n"}’ 40Gi FileSystemResizePending kubectl describe pvc pgdata-high –namespace databases | grep -A2 FileSystemResizePending FileSystemResizePending True Waiting for user to (re-)start a pod to finish file system resize of volume on node. # Inside the pod, nothing has changed yet kubectl exec pg-orders-0 –namespace databases — df -h /var/lib/postgresql/data Filesystem Size Used Avail Use% Mounted on /dev/sdb 20G 6.1G 13G 32% /var/lib/postgresql/data

Both numbers on screen are true at once. Control plane resize completed, so status.capacity says 40Gi. Node resize has not run, so the filesystem is still 20G. Restarting the consuming pod releases the second stage. For a StatefulSet that means deleting the pod and letting the controller recreate it, which is a real availability event and needs to be inside your change window rather than after it.

Verification, rollback, and a reclaim policy that decides both

# Finish the resize and verify green kubectl delete pod pg-orders-0 –namespace databases kubectl wait –for=condition=Ready pod/pg-orders-0 –namespace databases –timeout=300s pod/pg-orders-0 condition met kubectl exec pg-orders-0 –namespace databases — df -h /var/lib/postgresql/data Filesystem Size Used Avail Use% Mounted on /dev/sdb 40G 6.1G 32G 17% /var/lib/postgresql/data kubectl get pvc pgdata-high –namespace databases -o jsonpath='{.status.capacity.storage}{" "}{.status.allocatedResourceStatuses}{"n"}’ 40Gi # Static adoption verified end to end kubectl get pv static-archive-pv –namespace databases NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS AGE static-archive-pv 50Gi RWO Retain Bound databases/static-archive-pvc vsan-high-perf 6m

Green looks like three things together: status.capacity matching what you asked for, an empty allocatedResourceStatuses map, and df agreeing from inside the container. Any two of the three without the third means you stopped early. An empty allocatedResourceStatuses is the cleanest single signal, because that field only clears when both controller and node stages have finished.

Rollback deserves a blunt sentence: you cannot shrink a persistent volume. There is no undo for a size mistake, only forward motion and restores. Which means your rollback plan for anything in this Part is a volume snapshot taken before you started, exactly as built in Part 24 on VKS volume snapshots. Take it first, delete it after, and treat that as non negotiable for any database claim.

Where I disagree with the usual advice: Almost every tutorial leaves reclaimPolicy at Delete on the grounds that Kubernetes should tidy up after itself. On a VKS estate I set Retain on any class backing stateful production data, and I accept the orphaned volume cleanup work that creates. Reason: static provisioning has exactly one handle, the Supervisor PVC name, and a Delete policy destroys that handle the moment a workload cluster is torn down. Retain is what made archive-vol-01 available to adopt in Step 2 at all. The cost is real, roughly twenty minutes a month reconciling orphans against CNS in our estate, and it has already paid for itself twice.
Where expansion time actually goes Measured on a 3 zone VKS 3.3.1 lab, vSAN backed RWO claims. Seconds. Controller resize 20 to 40Gi 38 Node stage, offline, pod restart 80 Controller resize 40 to 80Gi 44 Node stage, online, no restart 12 Full 3 replica StatefulSet rollout 214 The disk grows in under a minute. Everything expensive is the pod lifecycle around it.

Plan expansion windows around pod restarts, not around disk size.

Choosing a provisioning mode, and a failure lookup

First table answers the objective directly. Objective 4.11 asks for strategies, and a strategy is a rule you can state before you see the manifest. Mine fits in six rows.

Situation Mode Reasoning
New StatefulSet, no pre existing dataDynamicPlatform owns the whole lifecycle, including deletion. Default answer.
Reusing data left behind by a Retain class after a cluster teardownStaticSupervisor PVC survived. Adopt it through volumeHandle.
Volume provisioned by a tenant through the VCF Automation Volume ServiceStaticVolume exists before any workload cluster claim does.
One volume shared by pods in two namespaces of the same clusterStaticDynamic provisioning binds one claim to one namespace.
One volume shared by two clusters in the same zoneStatic, PV and PVC authored in each clusterDocumented use case. Zone boundary still applies.
Claim must grow later without a maintenance windowDynamic, class with allowVolumeExpansion trueOnly dynamically provisioned claims are resizable.

Second table is the artifact worth bookmarking. Learn it by symptom, because symptoms are what an exam item and a broken cluster both hand you.

Symptom or error line Cause Remediation
storageclass.storage.k8s.io "vsan-high-perf" not foundPolicy assigned to the vSphere Namespace but class not listed in the cluster spec, or policy never assigned at all.Assign the policy in vCenter, then patch storageClasses in the Cluster object and wait for reconcile.
PVC Pending, no events at all, binding mode WaitForFirstConsumerWorking as designed. No pod has been scheduled yet.Schedule the consumer. Do not change the class.
only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resizeallowVolumeExpansion is false on the class, or the claim is statically provisioned.Set allowVolumeExpansion on the Supervisor class, then create a new claim. Existing claims are not retro enabled.
status.capacity shows the new size, df inside the pod shows the old sizeCondition FileSystemResizePending. Node stage has not run.Restart the consuming pod. For a StatefulSet, delete the pod and let the controller recreate it.
Static PV sits Available while its PVC sits PendingclaimRef namespace or name mismatch, capacity mismatch, or a different storageClassName on the two objects.Make capacity, accessModes, storageClassName and claimRef agree exactly, then reapply the claim.
Pod stuck ContainerCreating with an attach failure on a static volumeSupervisor PVC named in volumeHandle is still attached elsewhere, or nodeAffinity zone is missing on a zonal cluster.Detach the Supervisor side consumer, and add the topology.kubernetes.io/zone term for the zone holding the disk.
Resize request on a ReadWriteMany claim has no effectSupervisor does not support expansion for RWX volumes.Provision a new larger file share and migrate. Expansion is a block volume capability.
exceeded quota: databases-storagequota, requested storage, used, limitedvSphere Namespace storage limit reached, counted per storage policy.Raise the per policy limit on the vSphere Namespace, or reclaim orphaned volumes first.

Exam focus for objective 4.11

Exam focus, objective 4.11: Published wording is Create storage strategies using dynamic and static persistent volumes with support for expansion. What it expects you to be able to do: justify a provisioning mode from a described situation, author a static PersistentVolume that actually binds, and know what expansion does and does not cover. Item types this shape of objective tends to attract are matching, where a use case is paired with a provisioning mode; build list, where you order the steps of static adoption; multiple selection, where several expansion constraints are true and several are decoys; and point and click or hot area on the vSphere Client screen where a storage policy is assigned to a namespace. Trap that catches experienced admins: static provisioning muscle memory from vanilla vSphere CSI, where volumeHandle takes an FCD identifier. In a VKS cluster it takes the name of an unattached PVC on the Supervisor, in the vSphere Namespace hosting the cluster. Second trap, quieter: assuming that flipping allowVolumeExpansion to true makes existing claims resizable. It does not, and any answer built on editing a class to rescue a claim already in flight is wrong.
Objective checkpoint: Original questions written from the published objective wording. Nothing here reproduces or imitates a real item.

1. A tenant provisioned a 100Gi volume through the Volume Service. A platform engineer needs a pod in a VKS cluster to mount it. Which value belongs in spec.csi.volumeHandle, and what state must that object be in.
Answer: the name of the PersistentVolumeClaim created on the vSphere Namespace where the cluster is provisioned, and it must not be attached to any pod. An FCD identifier or the workload cluster PV name are both wrong.

2. A resize request is accepted, status.capacity reports the larger value, and the application still reports the old free space. Nothing is failing. What is happening and what clears it.
Answer: the controller stage finished and the node stage has not. The claim carries the FileSystemResizePending condition. Restarting the consuming pod lets kubelet complete the filesystem resize.

3. Two claims exist on a class where allowVolumeExpansion was set to true this morning. One was created last week, one an hour ago. Both are asked to grow. Which succeeds.
Answer: only the claim created after the class allowed expansion. Resizability is fixed at bind time, so the older claim is rejected as not resizable and must be replaced or restored into a larger claim.

Storage defaults I would standardise on

Field note, and it cost me a customer apology. Last winter we scheduled a 30 minute window to grow a PostgreSQL volume from 200Gi to 400Gi on a three replica StatefulSet. Patch applied in eleven seconds. Control plane reported 400Gi inside forty seconds. We checked the claim, saw the number we wanted, called the change complete and closed the window. Six hours later the database hit 99 percent on a filesystem that was still 200G, because none of the three pods had restarted and nobody had looked at the conditions. Recovery ran 95 minutes against a 30 minute plan, most of it spent doing an unplanned rolling restart during business hours with a nervous application team on a bridge. One jsonpath query would have caught it before we closed the window. That query is now a step in our runbook and it is the first thing I teach anyone preparing for this objective.

Verdict, plainly. Default to dynamic provisioning and treat static as an adoption tool rather than a design pattern; if your architecture needs static provisioning for new workloads, something upstream is wrong. Set allowVolumeExpansion true on every class you create, on day one, before any claim exists, because the retro enable path does not exist and the cost of the flag is zero. Set Retain on classes backing stateful production data and budget the orphan cleanup. Treat every expansion as a two stage operation with a pod restart in the plan, and put the snapshot from Part 24 in front of it. For the exam, spend your practice time on the static manifest rather than the dynamic one, because the dynamic path is four lines you already know and the static path has four fields that must agree.

Clean result checklist: Two storage policies assigned to the vSphere Namespace and both classes visible inside the workload cluster. A dynamic claim Bound on the high performance class. A static PersistentVolume Bound to its matching claim with volumeHandle pointing at an unattached Supervisor PVC and a zone affinity term present. One claim expanded end to end, with status.capacity, an empty allocatedResourceStatuses, and df inside the container all agreeing. A rejected resize captured from a class that forbids it, so you have seen the error text. A snapshot taken before the expansion and deleted after.

Tonight, in your own lab, do only this: create a claim on a class with allowVolumeExpansion set to false, try to grow it, and read the rejection out loud. Then create a second claim on a class that permits expansion, grow it, and stop before restarting the pod so you can watch status.capacity and df disagree. Fifteen minutes, two error states you will recognise instantly under time pressure. Next Part moves off the volume layer and onto workload deployment models. Namespace and zone mechanics behind all of this are in Part 17 if you need the ground underneath.

VCAP-VKS Exam Series · Part 26 of 34
« Previous: Part 25  |  Guide  |  Next: Part 27 »

References

Provision a Static Persistent Volume in a VKS Cluster, VMware Cloud Foundation 9.0 documentation
Persistent Volume Expansion for VKS Clusters, Broadcom TechDocs
Persistent Volumes, Kubernetes documentation, expanding claims and resize conditions
VCF 9.0 Volume Service, consuming static volumes via VKS, Cormac Hogan
VMware Certified Advanced Professional vSphere Kubernetes Service exam guide, 3V0-24.25

About The Author


Discover more from Journal of Intelligent Infrastructure

Subscribe to get the latest posts sent to your email.

Leave a Reply

Your email address will not be published. Required fields are marked *

Architect’s Toolkit

About the Author

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

Discover more from Journal of Intelligent Infrastructure

Subscribe now to keep reading and get access to the full archive.

Continue reading