Snapshot support that does not exist until you install it

Here is the line of output that reframes this whole objective. Same estate, same storage, same person at the keyboard, two contexts seconds apart.

That asymmetry is the objective. On a Supervisor, the pvCSI webhook and the external CSI snapshot webhook are installed for you, and a VolumeSnapshotClass exists from the moment a storage policy is bound to the vSphere Namespace. Inside a VKS cluster, none of it is there. Broadcom is explicit about it: both webhooks install automatically in Supervisor and must be deployed manually in VKS clusters. A candidate who has only ever snapshotted a Supervisor persistent volume will read the error above as a broken cluster, which is the single most expensive misread in this objective.

Trace it back to the Supervisor, as always in this series. Snapshot capability in a VKS cluster is a proxied capability. Your cluster does not talk to vSAN. It talks to pvCSI, which talks to the Supervisor, which talks to the Cloud Native Storage control plane in vCenter. Everything the cluster is allowed to do with storage was decided when somebody bound a storage policy to ns-platform-a. If that binding is wrong, no amount of package installation inside the cluster will produce a working snapshot, and the error you get will be about datastores rather than permissions.

flowchart TD
  A[Supervisor namespace] --> B[pvCSI webhook present by default]
  A --> C[CSI snapshot webhook present by default]
  A --> D[VolumeSnapshotClass exists on day one]
  E[VKS workload cluster] --> F[No snapshot CRDs until you act]
  F --> G[Install cert manager package]
  G --> H[Install external csi snapshot webhook]
  H --> I[Install vsphere pv csi webhook]
  I --> J[VolumeSnapshotClass appears]
  J --> K[VolumeSnapshot of a block PVC]
  K --> L[New PVC created through dataSource]

Webhook packages and a snapshot class you must not create

Prerequisites for this section are short and each one has a check that proves it. You need a VKr of v1.26.5 or later, because that is the floor Broadcom sets for CSI snapshot support on vSphere 8.0 Update 2 and later. You need a standard package repository of version v2023.9.19 or later reachable from the cluster, which Part 23 put into tkg-system. You need cert-manager installed, because both webhooks want serving certificates. Prove the first with a version read, prove the second with a package list, and only then install anything.

Notice what did not happen in that block. Nobody wrote a VolumeSnapshotClass. Every upstream Kubernetes CSI walkthrough on the internet opens by applying one, usually with a Retain deletion policy so that snapshots survive a careless delete, and that advice is wrong on VKS in a way that will cost you an item. Broadcom states it plainly: do not create volume snapshot classes, use only a pre-existing class, and classes with a Retain deletion policy are not supported. Your class arrives with the webhook, its policy is Delete, and that is the end of the discussion.

Dynamic snapshot of a PostgreSQL claim

Lab position: the pg-orders StatefulSet from Part 20 is running in namespace orders on vks-lifecycle-01, with a 100Gi ReadWriteOnce claim carved from the vsan-default policy. A dynamically provisioned snapshot is the common case and it is four lines of YAML. Quiesce your database first if you care about the restore being consistent, because a CSI snapshot is crash consistent and nothing more.

Nine seconds for a 100Gi volume, because nothing is copied. A snapshot on vSAN is metadata plus a redirect on write chain, and that is exactly why it is cheap to take and expensive to keep. Here are the four timings I measured on this estate, taken from a single loop of snapshot, restore, attach and clean up, so you can budget a maintenance window honestly instead of promising a minute and burning ten.

Two constraints govern how many of these you keep. Broadcom recommends two to three snapshots per virtual disk for performance, and caps vSAN ESA at 32 snapshots per volume. Read those numbers correctly. Thirty two is a ceiling that protects the platform, not a retention target, and storage quota monitoring does not account for snapshots at all, so a namespace can quietly consume capacity that no quota report will show you.

Restore paths and a deletion order that bites

Restore is where most people carry a wrong mental model in from the vSphere world. In vSphere you revert a virtual machine to a snapshot and the machine is as it was. Kubernetes has no equivalent. Upstream is unambiguous that the generally available snapshot implementation does not support reverting an existing PVC to an earlier state; you provision a new volume from the snapshot and you point your workload at it. That means a restore in a VKS cluster is always a new claim, a scale down, and a switch, never a single revert command.

Now the failure I promised, and it is the one Broadcom documents because it catches everybody. Somebody restores into vsan-highperf instead of vsan-default, reasoning that a recovery copy deserves the faster policy. Both policies are bound to ns-platform-a, both work for new claims, and the restore still fails, because a claim created from a snapshot must land on the same datastore as the source.

Backing out of a bad restore is simply deleting the new claim, since the source volume was never touched. Backing out of a stuck source volume is different and it has a strict order: every snapshot referencing that volume must go first, then expansion or deletion succeeds. A PVC sitting in Terminating forever, on a cluster where nothing else is wrong, is almost always a forgotten VolumeSnapshot in the same namespace.

Pre-provisioned snapshots are worth knowing about even though you will reach for them rarely. When a snapshot of the same block volume already exists on the Supervisor, you can surface it inside the VKS cluster by creating a VolumeSnapshotContent that references it and a VolumeSnapshot that binds to that content. That is the mechanism backup vendors use, and it is why third party integrations can present Supervisor level snapshots to cluster users without duplicating storage. Three ways of protecting the same volume, and they are not interchangeable.

Snapshot failure to remediation lookup

This table is the artifact from this Part worth keeping. Seven error strings, seven causes, seven fixes. Six of the seven are things the cluster tells you clearly if you read the event rather than the pod log, and every one of them has shown up in a real environment I have been called into.

Exam focus for objective 4.9

Snapshot practice I would standardise on

War story, and it ran for three weeks before anybody connected the dots. A platform team wrote a tidy little CronJob that snapshotted every database claim nightly and kept fourteen days of them. On vSAN OSA that meant fourteen snapshots per virtual disk against a documented recommendation of two to three. Write latency on their busiest PostgreSQL volume drifted from about 2 ms to 31 ms over those three weeks, slowly enough that everyone blamed the application. Then a capacity request came in, the volume needed to go from 200Gi to 400Gi, and the expansion was denied. That denial was the first hard signal anyone had. Deleting the backlog of snapshots took eleven minutes and brought write latency back under 3 ms the same evening. Three weeks of degraded database performance, caused entirely by a retention number somebody picked because it felt safe.

Verdict for this estate. Treat a VolumeSnapshot as a fifteen minute undo button and nothing longer. Take one immediately before a change, name it after the change rather than the date so its purpose is obvious, and delete it as part of the same change ticket. Pick two as your standing limit per virtual disk and enforce it, because three is the top of Broadcom guidance and fourteen is how you get the story above. Avoid the pattern of a scheduled snapshot CronJob entirely; it looks like backup, it is not backup, and it degrades exactly the volumes you care most about. For anything with a retention requirement measured in days, use Velero writing to object storage, which this series covers in Part 28. Snapshots and backups solve different problems and the exam expects you to keep them apart.

A clean result looks like this. Both webhook packages reconciled and their pods Running in kube-system. One VolumeSnapshotClass listed inside the cluster, driver csi.vsphere.vmware.com, deletion policy Delete, and none of it authored by you. A test VolumeSnapshot showing READYTOUSE true with a RESTORESIZE matching the source claim. A restore claim in Bound state built from that snapshot on the same storage class. A source volume that expands cleanly once the test snapshot is removed. If all five hold, objective 4.9 is in your hands rather than in your notes.

Tonight, in your own lab, do this. Take a snapshot of any block claim, then try to expand that claim and read the denial word for word. Delete the snapshot and run the same patch again. That thirty second loop is the whole operational half of this objective, and it is the fact most candidates learn during an outage instead. Component mechanics that this Part deliberately does not repeat live in the VKS series, and the full study path is indexed on the VCAP-VKS exam guide pillar. Part 25 moves up a layer, to upgrading the Supervisor Service itself.

References

• Creating Snapshots in a VKS Cluster, VCF 9.0, Broadcom TechDocs
• Create Snapshots in a VKS Cluster, VCF 9.0, Broadcom TechDocs
• Volume Snapshots, Kubernetes documentation
• VMware Cloud Foundation VKS Administrator Exam Guide, 3V0-24.25, Broadcom

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts