Where the Velero migration stands

One bucket decides this migration. Everything from here to the production cutover flows through a single S3 object store that a Velero install on the source writes to and a Velero install on the target reads from, and if that shared path is wrong, no workload moves cleanly. So before the first stateless service moves next part, this part builds and proves the pipe.

Here is the running example as it stands. Our source is a TKGI 1.18 estate on NSX-T with three clusters, dev, staging and prod, a Harbor registry, and one real stateful app, a web tier and an api tier in front of PostgreSQL with a 500 GB persistent volume, plus a couple of stateless services. Over the last five parts we stood up the target, a VKS cluster called dev-tkg-01 on vSphere Supervisor in VCF 9, with Antrea, vSphere CSI storage and Avi load balancing, and last part we translated the NSX-T distributed firewall rules into Antrea policy. Velero is what carries the workloads across that gap.

Velero (an open source backup and migration tool for Kubernetes, formerly Heptio Ark) does two jobs at once here. It captures the Kubernetes objects, the Deployments, Services, ConfigMaps and Secrets, as a JSON archive in object storage, and for stateful workloads it also moves the persistent volume data. That second job is where the method choice matters, and it is the part most people get wrong on vSphere.

Preflight on both clusters

Four things must be true before you install anything. You can reach both clusters with kubectl and switch contexts between them. Your source cluster runs Kubernetes 1.20 or later with a CSI driver that supports v1 volume snapshots, which TKGI 1.18 satisfies. An S3-compatible bucket exists and both clusters can route to its endpoint. And you have a credentials file for that bucket. Checks below prove the first three.

What green looks like: both contexts list, the source shows a CSI driver with a VolumeSnapshotClass, and the target shows at least one vSphere CSI storage class. Note the storage class names on both sides now, because a name that exists on the source and not on the target is the single most common reason a restore strands a database, and you want to catch it here rather than mid-migration.

This estate uses a storage class called tkgi-fast on the source; the target default is vsphere-csi-default. Those names differ, so hold that fact. It comes back to bite during verification, and the fix is in this part.

Install Velero on the source TKGI cluster

Install on the source first, because the source is where backups originate and where the CSI snapshots get taken. Two flags carry the weight. –features=EnableCSI turns on the CSI volume snapshot integration, which since Velero 1.14 ships inside the main binary, so there is no separate CSI plugin to install anymore. –use-node-agent installs the node-agent DaemonSet, the per-node component that runs the data mover pods that copy snapshot data to S3. Never put the bucket secret in the command line; keep it in a credentials file read from disk, as shown.

Give it a minute, then confirm the control plane deployment and the node-agent are both up. A node-agent pod per worker node is what you want; if the DaemonSet shows fewer ready than desired, data movement will stall later on the nodes that are missing an agent.

Install Velero on the target VKS cluster

Install on dev-tkg-01 with the same provider, plugin and bucket, so the target sees exactly the same backup repository the source writes. One change matters and it is not optional: mark the target BackupStorageLocation ReadOnly. A target cluster only needs to read backups and restore from them during migration. If both Velero installs treat the bucket as read-write, both run repository maintenance against it, and one cluster garbage collection can orphan or delete data the other still needs.

Within a minute the target Velero syncs the backup metadata from the bucket. Run velero backup get on dev-tkg-01 and you will see the backups the source created, even though nothing was taken on the target. That cross-cluster visibility is the whole point of sharing the bucket, and it is how a backup from tkgi-prod becomes restorable on dev-tkg-01. Size the bucket for the whole estate before you start, because three clusters, a 500 GB database plus smaller volumes, and a couple of full backups of retention headroom add up fast, and running out of object storage mid-wave stalls every in-flight DataUpload at once.

Choose a backup method for the migration

Velero offers three ways to move persistent volume data, and the obvious pick on vSphere is the wrong one for a migration. This decision table is the reference artifact for the series from here forward. Pin it, because every stateful workload you move gets classified by it, and the wrong row costs you either storage or time. File System Backup switched its default uploader to Kopia back in Velero 1.12 and dropped Restic in 1.15, so a current install already carries the faster mover, though it still reads the live file system and cannot promise the crash consistency a snapshot gives.

Table 1. Velero volume backup methods on VKS. Your migration lives in row one; row three is the trap.

flowchart LR
  A[Source TKGI cluster] --> B[Velero plus node-agent]
  B --> C[CSI snapshot of the PVC]
  C --> D[Data mover pod uploads]
  D --> E[(S3 object store)]
  E --> F[Target VKS Velero, read only]
  F --> G[Restore creates DataDownload]
  G --> H[New PVC on vSphere CSI]

Verify the toolchain, then roll it back

Prove the pipe with a real backup and restore before you trust it with a workload. Take a moving backup of the web-app namespace on the source with –snapshot-move-data, which is the flag that tells Velero to copy CSI snapshot data into the bucket rather than leave it as a local snapshot. Watch the DataUpload custom resource, the object Velero uses to drive one volume data copy, until it reaches Completed.

What green looks like: the backup phase reads Completed, not PartiallyFailed, and the DataUpload shows BYTES DONE equal to TOTAL BYTES. Velero polls that data movement every 10 seconds by default, and the whole operation has a 4 hour item operation timeout, so a 500 GB volume that finishes in minutes is well inside the envelope. Now switch to the target and restore into a scratch namespace to keep the check isolated.

On the target you can watch the mirror-image object, the DataDownload custom resource, which drives one volume restore the same way DataUpload drove the backup. Run kubectl -n velero get datadownloads and it walks from InProgress to Completed as the data mover streams the 500 GB volume back out of S3 and provisions a fresh PVC on vSphere CSI.

And here is the failure that shows up the first time nearly every estate runs this, the one the preflight warned about. That restore reports PartiallyFailed and the PostgreSQL PVC sits Pending forever, because Velero recreated it asking for storage class tkgi-fast, which lives on the source and not on dev-tkg-01.

A change-storage-class config map in the velero namespace on the target fixes it, rewriting the source class name to the target class during restore. Apply it once and it covers every future restore, so you never hand-edit a PVC.

Delete the failed restore, re-run it, and the PVC binds on vsphere-csi-default while the data mover streams the volume back from S3. Rollback of the toolchain itself is clean and first class, because nothing on the source was touched. A backup is a read of the source; the source cluster and its workloads keep running throughout. To back the toolchain out entirely, uninstall Velero on the target, and if you are abandoning the attempt, on the source too.

Table 2. Toolchain failures and their remediation. Rows one, two and five are the three that stop a migration cold.

Field note and verdict

Land the toolchain before the first wave

A clean result looks like this: Velero and a full set of node-agents Running on both clusters, the target BackupStorageLocation Available and ReadOnly, velero backup get on the target listing the backups the source created, a change-storage-class config map in place, the source repo secret copied over, and a throwaway namespace that backed up on tkgi-prod and restored onto dev-tkg-01 with its 500 GB volume intact. Hit that and the pipe is proven.

On your own estate on Monday, do one dry run before any real workload moves. Install Velero on a single source cluster and your first VKS cluster pointed at the same bucket, back up a disposable namespace with –snapshot-move-data, and restore it on the target into a scratch namespace. That one round trip surfaces every version, storage class and secret mismatch while the stakes are zero. For the storage design that sits under all of this, revisit the storage and data assessment from earlier in the series, and for how VKS provisions the clusters you are restoring onto, the VKS complete guide goes deeper than a migration runbook should. The full migration map lives on the TKGI to VKS guide. Next part moves the first stateless application across this pipe end to end.

Questions worth answering

Do I need the node-agent if I only back up stateless services?
No. Stateless backups capture Kubernetes objects only and run without a data mover. Install the node-agent anyway, because the stateful workloads are coming and CSI Snapshot Data Movement needs it.

Can both clusters write to the same bucket safely?
Only one should write. Keep the source read-write and the target read-only during migration. If you ever need the target to take its own backups, give it a separate bucket or prefix so the two repositories never overlap.

Where does the object storage come from on VKS?
Broadcom ships MinIO as a Supervisor Service you can enable for exactly this, an S3-compatible store the Supervisor hosts. Any S3-compatible endpoint both clusters can route to works; the bucket, not the vendor, is what matters.

Is File System Backup ever the right choice here?
Yes, for volumes CSI snapshot cannot take, NFS and file mode volumes. It reads the live file system so it is not crash consistent, so pair it with a backup hook that quiesces the application when the data matters.

This series covers a production migration. Run the dry run in a change window against your own environment, verify the restore on a scratch namespace, and leave the source clusters serving traffic until the target is proven.

References

• Backing Up and Restoring VKS Cluster Workloads, Broadcom TechDocs
• CSI Snapshot Data Movement, Velero documentation
• Restore Reference and changing storage classes, Velero documentation

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts