, ,

VKS Backup and Restore with Velero and External Object Storage (VCAP-VKS Exam Series, Part 28)

Objective 4.13 asks you to protect a VKS workload with Velero and an S3 bucket. This Part installs the Velero package from the VKS standard repository, moves a 5 Gi claim to MinIO with CSI snapshot data movement, and restores it into an emptied namespace.

VCAP-VKS Exam Series · Part 28 of 34
Who this is for: A candidate who took CSI volume snapshots in Part 24, worked the volume layer in Part 26, and now has to prove an application survives losing its whole namespace. This Part covers Objective 4.13, published wording Implement backup and restore strategies using Velero with external object storage [VERIFY the exact printed clause against the guide PDF linked in References; the objective number and scope are firm, my transcription of the sentence comes from a secondary listing]. Terms defined on first use: Velero is an open source backup tool for Kubernetes resources and volume data; an object store is S3 compatible bucket storage that lives outside the cluster, MinIO in this lab; a BackupStorageLocation, abbreviated BSL, is the Velero object naming that bucket; CSI is the Container Storage Interface, the driver layer that creates volumes on vSAN; a CSI snapshot is a point in time copy that stays on the same datastore; CSI snapshot data movement is the step that copies snapshot data out to the bucket; a DataUpload is the Velero custom resource that tracks one volume moving out; a DataDownload tracks one volume coming back; Kopia is the uploader Velero uses to write to the bucket; node-agent is the Velero daemonset that runs the data mover pods; VKS is vSphere Kubernetes Service, the product formerly named TKG Service or TKGS, and the rename is unfinished on VCF 9.0, which is why the package repository you are about to create lives in a namespace called tkg-system.
Key takeaways: Objective 4.13 is a restore objective wearing a backup objective costume. Velero on VKS is installed as a standard package through the VCF CLI, not with a raw velero install, and the piece that actually protects data is CSI snapshot data movement, which takes a vSAN snapshot and then copies it out to your bucket. Headline commands: velero backup create NAME –include-namespaces NS –wait and kubectl -n velero get datauploads -l velero.io/backup-name=NAME. Two numbers worth memorising: the node-agent gives a volume 30 minutes to provision before it cancels a DataUpload, and Velero gives the whole item operation 4 hours before it gives up. If you do not know both, a broken backup will look like a slow backup for most of a working day.

Of the eleven VCF 9 estates I have been inside this year, nine had Velero running on at least one VKS cluster. Three had ever completed a restore. That gap is not laziness. Installing Velero produces a satisfying green pod and a BackupStorageLocation that reports Available, and both of those are true statements about connectivity to a bucket rather than statements about your data being recoverable. Objective 4.13 is written by people who know the difference, which is why the verb in it is implement rather than install.

Data protection vocabulary for objective 4.13

Broadcom documents three ways for Velero to protect the data inside a persistent volume on a VKS cluster, and candidates routinely collapse them into one. They are not one. They differ in where the copy lands, how consistent it is, and whether the copy survives the loss of the datastore that holds the original. Pick the wrong one for a described scenario and every downstream answer in that item is wrong too.

Method Where the copy lands Consistency Use it on VKS when
CSI snapshot onlySame vSAN datastore as the source volumeCrash consistent point in timeNever as your only protection. Lose the datastore, lose both copies. This is the Part 24 mechanism, and it is a rollback tool.
CSI snapshot data movementYour S3 bucket, written by Kopia, snapshot deleted afterwardsCrash consistent, captured before the copy startsDefault choice for CNS block volumes on vSAN. This is what Broadcom recommends for VKS.
File system backupYour S3 bucket, read from the live mounted filesystemWeakest, files are read while the app writesOnly where no CSI snapshot exists, such as CNS file volumes, NFS mounts, emptyDir or local volumes.

Velero itself says CSI snapshot data movement should be preferred whenever it is available, because file system backup reads from a live volume and therefore never captures a single instant. On a vSAN backed VKS cluster it is always available, so file system backup on this estate is a fallback for a specific volume type rather than a strategy. That single sentence answers a surprising share of what objective 4.13 can ask.

Wall clock cost of each Velero operation One VKS 3.3.1 cluster, Kubernetes 1.32, a 5 Gi vSAN claim, MinIO on the same rack Metadata only, stateless app 11 s First data movement backup 70 s Second backup, same claim 24 s Namespace restore 29 s File system backup, same claim 218 s 0 s 230 s

Two things in that chart matter for design and both surprise people. First, restore is faster than backup on this estate, because a restore streams into a freshly provisioned volume while a backup has to snapshot, provision an intermediate claim, mount it and only then start reading. Second, the file system path costs roughly three times the data movement path on identical data. Anyone who reaches for file system backup because it sounds simpler is buying a slower and less consistent copy for no gain.

Prerequisites that decide whether a backup can even start

Last Part we put an application in front of users with an ingress controller and a private registry. This Part we take the same estate and remove the namespace on purpose. Before that is a defensible thing to do, four preflight facts have to be true, and every one of them was decided upstream of the VKS cluster you are logged into.

Trace it to the Supervisor: data movement provisions a temporary claim from the same StorageClass as the source volume. That StorageClass only exists inside the workload cluster because a storage policy was assigned to the vSphere Namespace on the Supervisor. Remove the policy assignment and the StorageClass vanishes from the cluster, the intermediate claim cannot bind, and your backup fails as a storage error 30 minutes later rather than as a permissions error immediately. Objective 4.13 failures are very often Part 7 failures with a delay attached.
# Tested against: VCF 9.0, Supervisor on Kubernetes 1.32, VKS 3.3.1, # kubectl v1.32.2, kubectl-vsphere plugin 9.0, VCF CLI 9.0.0, # Velero CLI v1.16.2_vmware.1, MinIO with a TLS certificate from the lab CA. $ kubectl config current-context pg-cluster-01 $ kubectl get nodes NAME STATUS ROLES AGE VERSION pg-cluster-01-ctrl-9x4mp Ready control-plane 6d v1.32.0+vmware.1-fips pg-cluster-01-np-a1-7c2kd-nx8lr Ready <none> 6d v1.32.0+vmware.1-fips pg-cluster-01-np-a1-7c2kd-q4vbt Ready <none> 6d v1.32.0+vmware.1-fips $ kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION vsan-default-storage-policy csi.vsphere.vmware.com Delete Immediate true vsan-high-perf-policy csi.vsphere.vmware.com Delete WaitForFirstConsumer true $ kubectl get volumesnapshotclasses NAME DRIVER DELETIONPOLICY AGE volumesnapshotclass-delete csi.vsphere.vmware.com Delete 6d $ velero version Client: Version: v1.16.2_vmware.1 <error getting server version: unable to retrieve the complete list of server APIs: velero.io/v1>

Read that last error rather than skipping it. It is the expected state before the server side exists, and it also tells you the CLI version you must match when you choose a package version. A client at 1.16.2 talking to a server at 1.17.0 is not a supported combination and it fails in ways that look like bucket problems.

Step by step, package repository to a verified restore

Step one is a values file, and it is where most installs go wrong. Velero needs a bucket, a region, path style addressing for MinIO, credentials, and if your object store presents a certificate from a private authority, that authority inline. Credentials never belong in a repository, so read them from the shell and render the file rather than typing secrets into YAML.

# Step 1. Build the values file. MINIO_KEY and MINIO_SECRET come from the # environment, never from the file you commit. $ cat > velero-data-values.yaml <<YAML backupStorageLocation: bucket: vks-velero-backups config: region: minio s3ForcePathStyle: "true" s3Url: https://minio.lab.local:9000 caCert: | $(sed ‘s/^/ /’ /etc/ssl/lab/lab-ca.crt) credential: | [default] aws_access_key_id=${MINIO_KEY} aws_secret_access_key=${MINIO_SECRET} YAML # Step 2. Add the VKS standard packages repository into tkg-system. The tkg # prefix is the old TKGS name showing through and is correct, do not change it. $ vcf-cli package repository add standard-package-repo –url projects.packages.broadcom.com/vsphere/supervisor/vks-standard-packages:3.3.0 -n tkg-system Waiting for package repository reconciliation for 'standard-package-repo' Fetch succeeded Template succeeded Deploy succeeded # Step 3. Confirm Velero is in the repository and pick the version that matches # your CLI. $ vcf-cli package available list -n tkg-system | grep velero velero.kubernetes.vmware.com velero $ vcf-cli package available get velero.kubernetes.vmware.com VERSION RELEASED-AT 1.16.1+vmware.1-vks.1 2025-05-19 12:30:00 +0000 UTC 1.16.2+vmware.1-vks.1 2025-08-05 12:30:00 +0000 UTC

Check the exact repository path in the VKS Standard Packages release notes for your VKS build before you run step two. Broadcom changes the dated path on every packages release, and a stale path fetches an older bundle that quietly lacks the Velero version you were told to install [VERIFY the 3.3.0 path above against the release notes linked in References; I verified the pattern and a later dated path, not this exact string, on the day of writing].

# Step 4. Install. Server components land in a namespace called velero, not in # tkg-system, which is where the PackageInstall lives. $ vcf-cli package install velero –namespace tkg-system –package velero.kubernetes.vmware.com –version 1.16.2+vmware.1-vks.1 –values-file velero-data-values.yaml Updating secret 'velero-tkg-system-values' Waiting for PackageInstall reconciliation for 'velero' Deploy succeeded # Step 5. Verify the server and, more importantly, the bucket. $ kubectl get pods -n velero NAME READY STATUS RESTARTS AGE node-agent-4jr7t 1/1 Running 0 2m11s node-agent-p9qzc 1/1 Running 0 2m11s velero-6c8f4d9b77-t2kxl 1/1 Running 0 2m11s $ kubectl get bsl -n velero NAME PHASE LAST VALIDATED AGE DEFAULT default Unavailable 14s 2m24s true # FAILURE, and the only place the reason is written down: $ kubectl -n velero logs deploy/velero | grep -i x509 | tail -1 level=error msg="Error getting backup store for this location" error="rpc error: code = Unknown desc = RequestError: send request failed caused by: Get "https://minio.lab.local:9000/vks-velero-backups?…": x509: certificate signed by unknown authority" # Cause: caCert in the values file was indented wrong, so the PEM was dropped. # Fix: correct the block, then reconcile. Do not reinstall. $ vcf-cli package installed update velero -n tkg-system –values-file velero-data-values.yaml $ kubectl get bsl -n velero NAME PHASE LAST VALIDATED AGE DEFAULT default Available 9s 6m02s true

A BackupStorageLocation reporting Available means Velero can list a bucket. It says nothing about whether a volume can be copied into it. Between those two facts sits a chain of six objects, and knowing the chain is the difference between reading a stalled backup in two minutes and staring at it for an afternoon.

flowchart TD
  A[velero backup create] --> B[Velero walks the namespace and finds a PVC]
  B --> C[CSI plugin creates VolumeSnapshot and VolumeSnapshotContent]
  C --> D[CSI plugin creates a DataUpload CR]
  D --> E{node agent claims the DataUpload}
  E -->|no pod on that node| F[Stalls in Accepted, cancelled after 30 minutes]
  E -->|claimed| G[Temporary PVC provisioned from the snapshot]
  G --> H{Bound within 30 minutes}
  H -->|no| F
  H -->|yes| I[Data mover pod runs Kopia and writes to the bucket]
  I --> J[DataUpload Completed, CSI snapshot deleted]
Object chain behind one volume in a Velero backup, with both places a VKS backup silently stalls
# Step 6. Back up one namespace, not the cluster. Our PostgreSQL StatefulSet # and its 5 Gi claim live in namespace shop-data. $ velero backup create shop-data-01 –include-namespaces shop-data –wait Backup request "shop-data-01" submitted successfully. Waiting for backup to complete. You may safely press ctrl-c to stop waiting. ……………………………………………………………. Backup completed with status: Completed. $ velero backup describe shop-data-01 Name: shop-data-01 Namespace: velero Phase: Completed Namespaces: Included: shop-data Storage Location: default Velero-Native Snapshot PVs: auto File System Backup (Default): false Snapshot Move Data: true Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 10m0s ItemOperationTimeout: 4h0m0s Started: 2026-06-04 09:12:41 +0000 UTC Completed: 2026-06-04 09:13:51 +0000 UTC Total items to be backed up: 58 Items backed up: 58 Backup Volumes: Velero-Native Snapshots: <none included> CSI Snapshots: shop-data/pgdata-shop-db-0: Data Movement: included Pod Volume Backups: <none included> # Green means these four lines together, not the word Completed on its own: # Snapshot Move Data: true, Data Mover: velero, # a CSI Snapshots entry naming your claim, Data Movement: included. $ kubectl -n velero get datauploads -l velero.io/backup-name=shop-data-01 NAME STATUS STARTED BYTES DONE TOTAL BYTES STORAGE LOCATION AGE shop-data-01-h6ftn Completed 70s 1379790848 1379790848 default 70s
Where the popular advice is wrong: Every tutorial you will find tells you to pass –snapshot-move-data on the backup command. On the VKS standard package that flag is already on, because the package sets it in the values, which is why the describe output above shows Snapshot Move Data true without it. The reverse case is what bites people. Install Velero by hand with a plain velero install, omit the flag, and Velero happily reports Completed while backing up nothing but Kubernetes objects. A metadata only backup of a database namespace restores an empty volume and reports success at every stage. Check the describe output, not the exit code.
# Step 7. Destroy the namespace, then restore it. Do this in a lab, tonight. $ kubectl delete ns shop-data namespace "shop-data" deleted $ kubectl get pv | grep shop-data (no output) $ velero restore create shop-data-r1 –from-backup shop-data-01 –wait Restore request "shop-data-r1" submitted successfully. Restore completed with status: Completed. $ velero restore describe shop-data-r1 Name: shop-data-r1 Phase: Completed Total items to be restored: 51 Items restored: 51 Started: 2026-06-04 09:31:08 +0000 UTC Completed: 2026-06-04 09:31:37 +0000 UTC Warnings: Namespaces: shop-data: could not restore, ConfigMap:kube-root-ca.crt already exists. Backup: shop-data-01 CSI Snapshot Restores: shop-data/pgdata-shop-db-0: Data Movement: specify –details for more information # Step 8. Verify the data, not the objects. A bound claim proves nothing. $ kubectl get pod,pvc -n shop-data NAME READY STATUS RESTARTS AGE pod/shop-db-0 1/1 Running 0 64s NAME STATUS VOLUME CAPACITY pvc/pgdata-shop-db-0 Bound pvc-3c81ab4e-6f20-4d1a-9e77-0b2c5a41d8e9 5Gi $ kubectl exec -n shop-data shop-db-0 — psql -U postgres -tAc 'select count(*) from orders' 148372 # 148372 rows, matching the pre delete count. That number is the restore test. # The kube-root-ca.crt warning is normal and always appears, ignore it.

Note what the restore did to the volume identity. Capacity, access mode and StorageClass came back identical, but the underlying PersistentVolume has a new name, because Velero provisioned a new volume and streamed data into it. Anything in your estate that referenced the old volume handle, a monitoring dashboard keyed on volume id or a static PV mapping, is now pointing at nothing. That is a real day two consequence and it does not appear in any Velero output.

Verification, rollback, retention and a failure lookup

Rollback on a backup system is unusual to think about, because a failed backup does not damage anything. What can damage things is a failed restore into a live namespace. Velero has no existing resource policy by default, which means it skips objects that already exist and leaves you with a half restored namespace and no warning louder than a line in describe. Restore into a new namespace first and switch traffic afterwards.

# Safe restore pattern. Land the backup somewhere new, prove it, then cut over. $ velero restore create shop-data-drill –from-backup shop-data-01 –namespace-mappings shop-data:shop-data-drill # Removing a backup and reclaiming its space are two different events. $ velero backup delete shop-data-01 –confirm $ velero repo get NAME STATUS LAST MAINTENANCE shop-data-default-kopia-2xg Ready 2026-06-04 03:00:12 +0000 UTC # Bucket usage does not drop until a full maintenance job runs. Watch that job. # SECOND FAILURE, the one that costs an evening. $ kubectl -n velero get datauploads -l velero.io/backup-name=shop-data-02 NAME STATUS STARTED BYTES DONE TOTAL BYTES NODE shop-data-02-k4ppl Accepted 38m <none> $ kubectl get pods -n velero -o wide | grep node-agent node-agent-4jr7t 1/1 Running 0 3d 10.244.1.7 pg-cluster-01-np-a1-7c2kd-nx8lr # Two worker nodes, one node-agent pod. The claim lives on the other node. $ kubectl describe daemonset node-agent -n velero | grep -A2 Events Warning FailedPlacement 0/3 nodes are available: 1 node had untolerated taint {workload=batch:NoSchedule} # Fix: add the toleration to the node-agent daemonset, or stop tainting the # node pool that holds your data.
Field note: I kicked off exactly that backup at 19:40 on a Thursday, saw Accepted, decided it was working and went home. It reported Failed at 23:47. Four hours and seven minutes to learn that a daemonset pod was missing, because the default item operation timeout is 4 hours and nothing before that moment prints a warning. I now run every lab and pre production Velero install with –item-operation-timeout set to 30 minutes, and I check kubectl get pods -n velero -o wide against kubectl get nodes before I trust a first backup on any cluster. Verdict: pick short timeouts and loud failures over long timeouts and polite silence. A backup that fails in half an hour is worth more than one that fails overnight.

Here is this Part's reference artifact, a symptom to owner lookup for Velero on VKS. Every row maps a string you will actually see to the object that owns the fix, and in three of six rows that object is not in the workload cluster at all.

What you see Real cause Where the fix lives
BSL PHASE Unavailable, x509 certificate signed by unknown authoritycaCert missing or mis indented in the values filevelero-data-values.yaml, then package installed update
DataUpload stuck in Accepted, NODE column emptyNo node-agent pod on the node holding the volume, usually a taintnode-agent daemonset tolerations in the velero namespace
DataUpload cancelled after 30 minutes, intermediate PVC PendingStorageClass gone or out of capacity, so the temporary claim never bindsStorage policy assignment on the vSphere Namespace, on the Supervisor
Backup Completed but Backup Volumes shows none includedData movement never engaged, metadata only backupBackup spec, snapshotMoveData, or the package values that set it
Restore Completed, pod CrashLoopBackOff, volume emptyRestored from a metadata only backup taken before the fix aboveNowhere. Take a new backup and prove it, the old one is not data
Failed to pull the Velero image during installPackage repository path is stale or the registry is not trustedRelease notes path, and the trusted CA config from Part 10

Exam focus for objective 4.13

Objective 4.13, what it actually tests: Whether you can choose a volume protection method for a described workload and justify it, whether you know which Velero object to inspect when a backup is not progressing, and whether you can sequence an install and a restore correctly. Expect it in scenario framed multiple choice, in build list items where you order the install and restore steps, in matching items pairing a Velero custom resource to what it tracks, and in point and click on a describe output where you identify the line proving volume data was included. The trap that catches experienced vSphere admins: they treat a snapshot as a backup because that instinct is correct for virtual machines and vSAN, where a snapshot plus array replication is a real strategy. On Kubernetes a CSI snapshot sits on the same datastore as the volume and is deleted by Velero as soon as data movement finishes. If an item describes protecting a workload against datastore loss and offers a snapshot only answer, that answer is wrong no matter how VMware shaped it looks.

Objective checkpoint

1. A VKS cluster holds an application whose data sits on a CNS file volume backed by NFS. Which Velero volume protection method applies? Answer: file system backup. Reasoning: CSI snapshot and CSI snapshot data movement both require a CSI driver that supports volume snapshots for that volume type, and CNS file volumes do not qualify, so reading the live filesystem is the only path.

2. A backup has been running for 45 minutes. velero backup describe shows Phase InProgress. Which single object tells you whether data is moving? Answer: the DataUpload custom resource in the velero namespace, filtered by the backup name label. Reasoning: DataUpload carries the phase, the node that accepted it and the bytes transferred, none of which appear in the backup object until the operation reaches a terminal state.

3. You must restore a namespace that still exists and is serving traffic, without disturbing it. Which restore option do you use? Answer: –namespace-mappings to land the restore in a new namespace. Reasoning: restoring over a live namespace skips existing objects by default and produces a mixed state, so a mapped restore proves the backup while leaving production untouched.

Protection defaults worth committing to

My recommendation for this estate is narrow and I would defend it in a design review. Install Velero from the VKS standard packages repository rather than upstream, so version compatibility is Broadcom's problem rather than yours. Use CSI snapshot data movement everywhere, and treat file system backup as an exception you have to justify per volume. Back up one namespace per backup rather than the whole cluster, because Velero creates a backup repository per namespace anyway and a cluster wide backup sweeps in every package and package metadata object that the standard repository installed, which inflates item counts and hands you stale package objects on restore. Set item operation timeout to something you will actually wait for. And write down, next to the runbook, the row count or file hash that proves a restore worked, because a bound claim is not evidence.

What a clean result looks like: node-agent running on every worker node that can hold a volume, one BackupStorageLocation in phase Available marked default, a backup whose describe output names your claim under CSI Snapshots with Data Movement included, a DataUpload in Completed with a non zero bytes count, a restore into a mapped namespace that comes back with your row count intact, and a repository maintenance job that has run inside the last 24 hours. If any one of those six is missing you have a backup system that reports success.

Tonight, in your own lab: take one backup of a namespace that has data in it, then restore it with –namespace-mappings into a second namespace and compare a row count or a checksum across the two. It costs about five minutes on a small claim, it destroys nothing, and it is the single exercise that turns objective 4.13 from something you have read about into something you have done. Next Part moves from protecting a deployment to editing one, where a single YAML field decides whether a VKS cluster reconciles or wedges.

VCAP-VKS Exam Series · Part 28 of 34
« Previous: Part 27  |  Guide  |  Next: Part 29 »

References

Product mechanics for VKS itself are covered in the VKS Series, and the registry that feeds these packages is covered in the Harbor guide. Readers arriving from a Tanzu Kubernetes Grid Integrated estate should start at the TKGI to VKS Series instead.

About The Author


Discover more from Journal of Intelligent Infrastructure

Subscribe to get the latest posts sent to your email.

Leave a Reply

Your email address will not be published. Required fields are marked *

Architect’s Toolkit

About the Author

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

Discover more from Journal of Intelligent Infrastructure

Subscribe now to keep reading and get access to the full archive.

Continue reading