VKS Backup and Restore with Velero and External Object Storage (VCAP-VKS Exam Series, Part 28)
Objective 4.13 asks you to protect a VKS workload with Velero and an S3 bucket. This Part installs the Velero package from the VKS standard repository, moves a 5 Gi claim to MinIO with CSI snapshot data movement, and restores it into an emptied namespace.
Who this is for: A candidate who took CSI volume snapshots in Part 24, worked the volume layer in Part 26, and now has to prove an application survives losing its whole namespace. This Part covers Objective 4.13, published wording Implement backup and restore strategies using Velero with external object storage [VERIFY the exact printed clause against the guide PDF linked in References; the objective number and scope are firm, my transcription of the sentence comes from a secondary listing]. Terms defined on first use: Velero is an open source backup tool for Kubernetes resources and volume data; an object store is S3 compatible bucket storage that lives outside the cluster, MinIO in this lab; a BackupStorageLocation, abbreviated BSL, is the Velero object naming that bucket; CSI is the Container Storage Interface, the driver layer that creates volumes on vSAN; a CSI snapshot is a point in time copy that stays on the same datastore; CSI snapshot data movement is the step that copies snapshot data out to the bucket; a DataUpload is the Velero custom resource that tracks one volume moving out; a DataDownload tracks one volume coming back; Kopia is the uploader Velero uses to write to the bucket; node-agent is the Velero daemonset that runs the data mover pods; VKS is vSphere Kubernetes Service, the product formerly named TKG Service or TKGS, and the rename is unfinished on VCF 9.0, which is why the package repository you are about to create lives in a namespace called tkg-system.
Key takeaways: Objective 4.13 is a restore objective wearing a backup objective costume. Velero on VKS is installed as a standard package through the VCF CLI, not with a raw velero install, and the piece that actually protects data is CSI snapshot data movement, which takes a vSAN snapshot and then copies it out to your bucket. Headline commands: velero backup create NAME –include-namespaces NS –wait and kubectl -n velero get datauploads -l velero.io/backup-name=NAME. Two numbers worth memorising: the node-agent gives a volume 30 minutes to provision before it cancels a DataUpload, and Velero gives the whole item operation 4 hours before it gives up. If you do not know both, a broken backup will look like a slow backup for most of a working day.
Of the eleven VCF 9 estates I have been inside this year, nine had Velero running on at least one VKS cluster. Three had ever completed a restore. That gap is not laziness. Installing Velero produces a satisfying green pod and a BackupStorageLocation that reports Available, and both of those are true statements about connectivity to a bucket rather than statements about your data being recoverable. Objective 4.13 is written by people who know the difference, which is why the verb in it is implement rather than install.
Data protection vocabulary for objective 4.13
Broadcom documents three ways for Velero to protect the data inside a persistent volume on a VKS cluster, and candidates routinely collapse them into one. They are not one. They differ in where the copy lands, how consistent it is, and whether the copy survives the loss of the datastore that holds the original. Pick the wrong one for a described scenario and every downstream answer in that item is wrong too.
Method
Where the copy lands
Consistency
Use it on VKS when
CSI snapshot only
Same vSAN datastore as the source volume
Crash consistent point in time
Never as your only protection. Lose the datastore, lose both copies. This is the Part 24 mechanism, and it is a rollback tool.
CSI snapshot data movement
Your S3 bucket, written by Kopia, snapshot deleted afterwards
Crash consistent, captured before the copy starts
Default choice for CNS block volumes on vSAN. This is what Broadcom recommends for VKS.
File system backup
Your S3 bucket, read from the live mounted filesystem
Weakest, files are read while the app writes
Only where no CSI snapshot exists, such as CNS file volumes, NFS mounts, emptyDir or local volumes.
Velero itself says CSI snapshot data movement should be preferred whenever it is available, because file system backup reads from a live volume and therefore never captures a single instant. On a vSAN backed VKS cluster it is always available, so file system backup on this estate is a fallback for a specific volume type rather than a strategy. That single sentence answers a surprising share of what objective 4.13 can ask.
Two things in that chart matter for design and both surprise people. First, restore is faster than backup on this estate, because a restore streams into a freshly provisioned volume while a backup has to snapshot, provision an intermediate claim, mount it and only then start reading. Second, the file system path costs roughly three times the data movement path on identical data. Anyone who reaches for file system backup because it sounds simpler is buying a slower and less consistent copy for no gain.
Prerequisites that decide whether a backup can even start
Last Part we put an application in front of users with an ingress controller and a private registry. This Part we take the same estate and remove the namespace on purpose. Before that is a defensible thing to do, four preflight facts have to be true, and every one of them was decided upstream of the VKS cluster you are logged into.
Trace it to the Supervisor: data movement provisions a temporary claim from the same StorageClass as the source volume. That StorageClass only exists inside the workload cluster because a storage policy was assigned to the vSphere Namespace on the Supervisor. Remove the policy assignment and the StorageClass vanishes from the cluster, the intermediate claim cannot bind, and your backup fails as a storage error 30 minutes later rather than as a permissions error immediately. Objective 4.13 failures are very often Part 7 failures with a delay attached.
# Tested against: VCF 9.0, Supervisor on Kubernetes 1.32, VKS 3.3.1,
# kubectl v1.32.2, kubectl-vsphere plugin 9.0, VCF CLI 9.0.0,
# Velero CLI v1.16.2_vmware.1, MinIO with a TLS certificate from the lab CA.
$ kubectl config current-context
pg-cluster-01
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
pg-cluster-01-ctrl-9x4mp Ready control-plane 6d v1.32.0+vmware.1-fips
pg-cluster-01-np-a1-7c2kd-nx8lr Ready <none> 6d v1.32.0+vmware.1-fips
pg-cluster-01-np-a1-7c2kd-q4vbt Ready <none> 6d v1.32.0+vmware.1-fips
$ kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION
vsan-default-storage-policy csi.vsphere.vmware.com Delete Immediate true
vsan-high-perf-policy csi.vsphere.vmware.com Delete WaitForFirstConsumer true
$ kubectl get volumesnapshotclasses
NAME DRIVER DELETIONPOLICY AGE
volumesnapshotclass-delete csi.vsphere.vmware.com Delete 6d
$ velero version
Client:
Version: v1.16.2_vmware.1
<error getting server version: unable to retrieve the complete list of server APIs: velero.io/v1>
Read that last error rather than skipping it. It is the expected state before the server side exists, and it also tells you the CLI version you must match when you choose a package version. A client at 1.16.2 talking to a server at 1.17.0 is not a supported combination and it fails in ways that look like bucket problems.
Step by step, package repository to a verified restore
Step one is a values file, and it is where most installs go wrong. Velero needs a bucket, a region, path style addressing for MinIO, credentials, and if your object store presents a certificate from a private authority, that authority inline. Credentials never belong in a repository, so read them from the shell and render the file rather than typing secrets into YAML.
# Step 1. Build the values file. MINIO_KEY and MINIO_SECRET come from the
# environment, never from the file you commit.
$ cat > velero-data-values.yaml <<YAML
backupStorageLocation:
bucket: vks-velero-backups
config:
region: minio
s3ForcePathStyle: "true"
s3Url: https://minio.lab.local:9000
caCert: |
$(sed ‘s/^/ /’ /etc/ssl/lab/lab-ca.crt)
credential: |
[default]
aws_access_key_id=${MINIO_KEY}
aws_secret_access_key=${MINIO_SECRET}
YAML
# Step 2. Add the VKS standard packages repository into tkg-system. The tkg
# prefix is the old TKGS name showing through and is correct, do not change it.
$ vcf-cli package repository add standard-package-repo
–url projects.packages.broadcom.com/vsphere/supervisor/vks-standard-packages:3.3.0
-n tkg-system
Waiting for package repository reconciliation for 'standard-package-repo'
Fetch succeeded
Template succeeded
Deploy succeeded
# Step 3. Confirm Velero is in the repository and pick the version that matches
# your CLI.
$ vcf-cli package available list -n tkg-system | grep velero
velero.kubernetes.vmware.com velero
$ vcf-cli package available get velero.kubernetes.vmware.com
VERSION RELEASED-AT
1.16.1+vmware.1-vks.1 2025-05-19 12:30:00 +0000 UTC
1.16.2+vmware.1-vks.1 2025-08-05 12:30:00 +0000 UTC
Check the exact repository path in the VKS Standard Packages release notes for your VKS build before you run step two. Broadcom changes the dated path on every packages release, and a stale path fetches an older bundle that quietly lacks the Velero version you were told to install [VERIFY the 3.3.0 path above against the release notes linked in References; I verified the pattern and a later dated path, not this exact string, on the day of writing].
# Step 4. Install. Server components land in a namespace called velero, not in
# tkg-system, which is where the PackageInstall lives.
$ vcf-cli package install velero –namespace tkg-system
–package velero.kubernetes.vmware.com
–version 1.16.2+vmware.1-vks.1
–values-file velero-data-values.yaml
Updating secret 'velero-tkg-system-values'
Waiting for PackageInstall reconciliation for 'velero'
Deploy succeeded
# Step 5. Verify the server and, more importantly, the bucket.
$ kubectl get pods -n velero
NAME READY STATUS RESTARTS AGE
node-agent-4jr7t 1/1 Running 0 2m11s
node-agent-p9qzc 1/1 Running 0 2m11s
velero-6c8f4d9b77-t2kxl 1/1 Running 0 2m11s
$ kubectl get bsl -n velero
NAME PHASE LAST VALIDATED AGE DEFAULT
default Unavailable 14s 2m24s true
# FAILURE, and the only place the reason is written down:
$ kubectl -n velero logs deploy/velero | grep -i x509 | tail -1
level=error msg="Error getting backup store for this location"
error="rpc error: code = Unknown desc = RequestError: send request failed
caused by: Get "https://minio.lab.local:9000/vks-velero-backups?…":
x509: certificate signed by unknown authority"
# Cause: caCert in the values file was indented wrong, so the PEM was dropped.
# Fix: correct the block, then reconcile. Do not reinstall.
$ vcf-cli package installed update velero -n tkg-system –values-file velero-data-values.yaml
$ kubectl get bsl -n velero
NAME PHASE LAST VALIDATED AGE DEFAULT
default Available 9s 6m02s true
A BackupStorageLocation reporting Available means Velero can list a bucket. It says nothing about whether a volume can be copied into it. Between those two facts sits a chain of six objects, and knowing the chain is the difference between reading a stalled backup in two minutes and staring at it for an afternoon.
flowchart TD
A[velero backup create] --> B[Velero walks the namespace and finds a PVC]
B --> C[CSI plugin creates VolumeSnapshot and VolumeSnapshotContent]
C --> D[CSI plugin creates a DataUpload CR]
D --> E{node agent claims the DataUpload}
E -->|no pod on that node| F[Stalls in Accepted, cancelled after 30 minutes]
E -->|claimed| G[Temporary PVC provisioned from the snapshot]
G --> H{Bound within 30 minutes}
H -->|no| F
H -->|yes| I[Data mover pod runs Kopia and writes to the bucket]
I --> J[DataUpload Completed, CSI snapshot deleted]
Object chain behind one volume in a Velero backup, with both places a VKS backup silently stalls
# Step 6. Back up one namespace, not the cluster. Our PostgreSQL StatefulSet
# and its 5 Gi claim live in namespace shop-data.
$ velero backup create shop-data-01 –include-namespaces shop-data –wait
Backup request "shop-data-01" submitted successfully.
Waiting for backup to complete. You may safely press ctrl-c to stop waiting.
…………………………………………………………….
Backup completed with status: Completed.
$ velero backup describe shop-data-01
Name: shop-data-01
Namespace: velero
Phase: Completed
Namespaces:
Included: shop-data
Storage Location: default
Velero-Native Snapshot PVs: auto
File System Backup (Default): false
Snapshot Move Data: true
Data Mover: velero
TTL: 720h0m0s
CSISnapshotTimeout: 10m0s
ItemOperationTimeout: 4h0m0s
Started: 2026-06-04 09:12:41 +0000 UTC
Completed: 2026-06-04 09:13:51 +0000 UTC
Total items to be backed up: 58
Items backed up: 58
Backup Volumes:
Velero-Native Snapshots: <none included>
CSI Snapshots:
shop-data/pgdata-shop-db-0:
Data Movement: included
Pod Volume Backups: <none included>
# Green means these four lines together, not the word Completed on its own:
# Snapshot Move Data: true, Data Mover: velero,
# a CSI Snapshots entry naming your claim, Data Movement: included.
$ kubectl -n velero get datauploads -l velero.io/backup-name=shop-data-01
NAME STATUS STARTED BYTES DONE TOTAL BYTES STORAGE LOCATION AGE
shop-data-01-h6ftn Completed 70s 1379790848 1379790848 default 70s
Where the popular advice is wrong: Every tutorial you will find tells you to pass –snapshot-move-data on the backup command. On the VKS standard package that flag is already on, because the package sets it in the values, which is why the describe output above shows Snapshot Move Data true without it. The reverse case is what bites people. Install Velero by hand with a plain velero install, omit the flag, and Velero happily reports Completed while backing up nothing but Kubernetes objects. A metadata only backup of a database namespace restores an empty volume and reports success at every stage. Check the describe output, not the exit code.
# Step 7. Destroy the namespace, then restore it. Do this in a lab, tonight.
$ kubectl delete ns shop-data
namespace "shop-data" deleted
$ kubectl get pv | grep shop-data
(no output)
$ velero restore create shop-data-r1 –from-backup shop-data-01 –wait
Restore request "shop-data-r1" submitted successfully.
Restore completed with status: Completed.
$ velero restore describe shop-data-r1
Name: shop-data-r1
Phase: Completed
Total items to be restored: 51
Items restored: 51
Started: 2026-06-04 09:31:08 +0000 UTC
Completed: 2026-06-04 09:31:37 +0000 UTC
Warnings:
Namespaces:
shop-data: could not restore, ConfigMap:kube-root-ca.crt already exists.
Backup: shop-data-01
CSI Snapshot Restores:
shop-data/pgdata-shop-db-0:
Data Movement: specify –details for more information
# Step 8. Verify the data, not the objects. A bound claim proves nothing.
$ kubectl get pod,pvc -n shop-data
NAME READY STATUS RESTARTS AGE
pod/shop-db-0 1/1 Running 0 64s
NAME STATUS VOLUME CAPACITY
pvc/pgdata-shop-db-0 Bound pvc-3c81ab4e-6f20-4d1a-9e77-0b2c5a41d8e9 5Gi
$ kubectl exec -n shop-data shop-db-0 — psql -U postgres -tAc
'select count(*) from orders'
148372
# 148372 rows, matching the pre delete count. That number is the restore test.
# The kube-root-ca.crt warning is normal and always appears, ignore it.
Note what the restore did to the volume identity. Capacity, access mode and StorageClass came back identical, but the underlying PersistentVolume has a new name, because Velero provisioned a new volume and streamed data into it. Anything in your estate that referenced the old volume handle, a monitoring dashboard keyed on volume id or a static PV mapping, is now pointing at nothing. That is a real day two consequence and it does not appear in any Velero output.
Verification, rollback, retention and a failure lookup
Rollback on a backup system is unusual to think about, because a failed backup does not damage anything. What can damage things is a failed restore into a live namespace. Velero has no existing resource policy by default, which means it skips objects that already exist and leaves you with a half restored namespace and no warning louder than a line in describe. Restore into a new namespace first and switch traffic afterwards.
# Safe restore pattern. Land the backup somewhere new, prove it, then cut over.
$ velero restore create shop-data-drill
–from-backup shop-data-01
–namespace-mappings shop-data:shop-data-drill
# Removing a backup and reclaiming its space are two different events.
$ velero backup delete shop-data-01 –confirm
$ velero repo get
NAME STATUS LAST MAINTENANCE
shop-data-default-kopia-2xg Ready 2026-06-04 03:00:12 +0000 UTC
# Bucket usage does not drop until a full maintenance job runs. Watch that job.
# SECOND FAILURE, the one that costs an evening.
$ kubectl -n velero get datauploads -l velero.io/backup-name=shop-data-02
NAME STATUS STARTED BYTES DONE TOTAL BYTES NODE
shop-data-02-k4ppl Accepted 38m <none>
$ kubectl get pods -n velero -o wide | grep node-agent
node-agent-4jr7t 1/1 Running 0 3d 10.244.1.7 pg-cluster-01-np-a1-7c2kd-nx8lr
# Two worker nodes, one node-agent pod. The claim lives on the other node.
$ kubectl describe daemonset node-agent -n velero | grep -A2 Events
Warning FailedPlacement 0/3 nodes are available: 1 node had untolerated
taint {workload=batch:NoSchedule}
# Fix: add the toleration to the node-agent daemonset, or stop tainting the
# node pool that holds your data.
Field note: I kicked off exactly that backup at 19:40 on a Thursday, saw Accepted, decided it was working and went home. It reported Failed at 23:47. Four hours and seven minutes to learn that a daemonset pod was missing, because the default item operation timeout is 4 hours and nothing before that moment prints a warning. I now run every lab and pre production Velero install with –item-operation-timeout set to 30 minutes, and I check kubectl get pods -n velero -o wide against kubectl get nodes before I trust a first backup on any cluster. Verdict: pick short timeouts and loud failures over long timeouts and polite silence. A backup that fails in half an hour is worth more than one that fails overnight.
Here is this Part's reference artifact, a symptom to owner lookup for Velero on VKS. Every row maps a string you will actually see to the object that owns the fix, and in three of six rows that object is not in the workload cluster at all.
What you see
Real cause
Where the fix lives
BSL PHASE Unavailable, x509 certificate signed by unknown authority
caCert missing or mis indented in the values file
velero-data-values.yaml, then package installed update
DataUpload stuck in Accepted, NODE column empty
No node-agent pod on the node holding the volume, usually a taint
node-agent daemonset tolerations in the velero namespace
DataUpload cancelled after 30 minutes, intermediate PVC Pending
StorageClass gone or out of capacity, so the temporary claim never binds
Storage policy assignment on the vSphere Namespace, on the Supervisor
Backup Completed but Backup Volumes shows none included
Data movement never engaged, metadata only backup
Backup spec, snapshotMoveData, or the package values that set it
Restore Completed, pod CrashLoopBackOff, volume empty
Restored from a metadata only backup taken before the fix above
Nowhere. Take a new backup and prove it, the old one is not data
Failed to pull the Velero image during install
Package repository path is stale or the registry is not trusted
Release notes path, and the trusted CA config from Part 10
Exam focus for objective 4.13
Objective 4.13, what it actually tests: Whether you can choose a volume protection method for a described workload and justify it, whether you know which Velero object to inspect when a backup is not progressing, and whether you can sequence an install and a restore correctly. Expect it in scenario framed multiple choice, in build list items where you order the install and restore steps, in matching items pairing a Velero custom resource to what it tracks, and in point and click on a describe output where you identify the line proving volume data was included. The trap that catches experienced vSphere admins: they treat a snapshot as a backup because that instinct is correct for virtual machines and vSAN, where a snapshot plus array replication is a real strategy. On Kubernetes a CSI snapshot sits on the same datastore as the volume and is deleted by Velero as soon as data movement finishes. If an item describes protecting a workload against datastore loss and offers a snapshot only answer, that answer is wrong no matter how VMware shaped it looks.
Objective checkpoint
1. A VKS cluster holds an application whose data sits on a CNS file volume backed by NFS. Which Velero volume protection method applies? Answer: file system backup. Reasoning: CSI snapshot and CSI snapshot data movement both require a CSI driver that supports volume snapshots for that volume type, and CNS file volumes do not qualify, so reading the live filesystem is the only path.
2. A backup has been running for 45 minutes. velero backup describe shows Phase InProgress. Which single object tells you whether data is moving? Answer: the DataUpload custom resource in the velero namespace, filtered by the backup name label. Reasoning: DataUpload carries the phase, the node that accepted it and the bytes transferred, none of which appear in the backup object until the operation reaches a terminal state.
3. You must restore a namespace that still exists and is serving traffic, without disturbing it. Which restore option do you use? Answer: –namespace-mappings to land the restore in a new namespace. Reasoning: restoring over a live namespace skips existing objects by default and produces a mixed state, so a mapped restore proves the backup while leaving production untouched.
Protection defaults worth committing to
My recommendation for this estate is narrow and I would defend it in a design review. Install Velero from the VKS standard packages repository rather than upstream, so version compatibility is Broadcom's problem rather than yours. Use CSI snapshot data movement everywhere, and treat file system backup as an exception you have to justify per volume. Back up one namespace per backup rather than the whole cluster, because Velero creates a backup repository per namespace anyway and a cluster wide backup sweeps in every package and package metadata object that the standard repository installed, which inflates item counts and hands you stale package objects on restore. Set item operation timeout to something you will actually wait for. And write down, next to the runbook, the row count or file hash that proves a restore worked, because a bound claim is not evidence.
What a clean result looks like: node-agent running on every worker node that can hold a volume, one BackupStorageLocation in phase Available marked default, a backup whose describe output names your claim under CSI Snapshots with Data Movement included, a DataUpload in Completed with a non zero bytes count, a restore into a mapped namespace that comes back with your row count intact, and a repository maintenance job that has run inside the last 24 hours. If any one of those six is missing you have a backup system that reports success.
Tonight, in your own lab: take one backup of a namespace that has data in it, then restore it with –namespace-mappings into a second namespace and compare a row count or a checksum across the two. It costs about five minutes on a small claim, it destroys nothing, and it is the single exercise that turns objective 4.13 from something you have read about into something you have done. Next Part moves from protecting a deployment to editing one, where a single YAML field decides whether a VKS cluster reconciles or wedges.
Product mechanics for VKS itself are covered in the VKS Series, and the registry that feeds these packages is covered in the Harbor guide. Readers arriving from a Tanzu Kubernetes Grid Integrated estate should start at the TKGI to VKS Series instead.
Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.
Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.
This site uses cookies only for anonymous analytics, to understand which guides and tools are useful. Accept to help, or decline — either way the site works fully.