Five hundred gigabytes copied cleanly. Eight kilobytes of it, the pg_control file, arrived mid-write, and that was enough to make PostgreSQL refuse to start on the target. Its web tier moved like the stateless service in the last part, quick and boring. A database volume is where a migration earns its scars, because a volume is not just bytes, it is bytes captured at a single instant, and getting that instant wrong turns 430 GB of good data into a cluster that logs could not locate a valid checkpoint record and sits in CrashLoopBackOff.

Position going into the stateful move

Last part carried storefront, a stateless catalog frontend, from the TKGI cluster tkgi-prod onto the VKS cluster dev-tkg-01 through a shared S3 bucket, and the only real work was clearing two admission gates. This part moves the webapp namespace, which is the harder half of the running example: a web and api tier in front of a PostgreSQL 15 database running as a StatefulSet named postgres, with one persistent volume claim, data-postgres-0, provisioned at 500 GB on the source. Of that, 430 GB is in use, and the orders table alone holds 6.2 million rows. Its web and api tier behave exactly like storefront did, so I will not relitigate Pod Security here. That volume is the whole story.

Two terms before the procedure. File System Backup, FSB, reads a volume file by file from the live filesystem while the application keeps writing. CSI Snapshot Data Movement, driven by the –snapshot-move-data flag, first asks the Container Storage Interface driver for a point-in-time snapshot, then uses Velero built-in Kopia uploader to move that snapshot data to the shared bucket. Part 17 installed both the node agent and the data mover on each cluster, so both paths are available. For a large, busy database the difference between them is the difference between a clean restore and a wasted afternoon.

Preflight for a stateful cutover

Four checks earn this backup, and skipping any one of them is how the restore fails an hour later when you have stopped watching. Both clusters must run the node agent, because snapshot data movement uses it to run Kopia. Your source must have a VolumeSnapshotClass wired to its vSphere CSI driver, or the snapshot step has nothing to call. You need the exact storage class names on each side, because you are about to map one to the other. And you need to know the database is healthy before you freeze it, so you are not migrating a problem.

What green looks like: both node-agent daemonsets report desired equal to ready, the source lists a VolumeSnapshotClass, and the two get sc outputs disagree on names, which is the point. Here the source shows tkgi-vsan-default as default and the target shows vks-vsan-default. Write both down, because the next step depends on the exact strings.

Map the storage class before restore

This is the step people discover the hard way, so do it first. Velero restores a PVC with the storageClassName it was backed up with. On the target that name, tkgi-vsan-default, does not exist, so the PVC has no provisioner willing to claim it and sits Pending forever while the postgres pod waits on a volume that will never bind. Fixing it takes a ConfigMap in the velero namespace on the target that maps the old class name to the new one. Velero built-in change-storage-class restore action reads it by label, not by name, so the labels must be exact.

Apply it with kubectl against the target, confirm it lands, and treat this ConfigMap as the reference artifact for the whole database wave. Every stateful namespace you move gets a data row here, one line per source class you are retiring. Pin the table below next to it.

Table 1. Storage class mapping for the database wave. This is the reference artifact, one row per retiring class, and it drives the ConfigMap above.

Quiesce PostgreSQL and move the volume

A CSI snapshot freezes the block device at one instant, which is crash consistent, the same state the volume would be in after a power cut. PostgreSQL is built to recover from a power cut by replaying its write-ahead log, so crash consistent is usually enough. Usually is not a word I ship a production database on. A pre-backup hook that runs CHECKPOINT flushes dirty buffers to disk and writes a clean restart point, which shortens replay and, more importantly, guarantees the snapshot lands on a checkpoint boundary rather than mid-flush. Annotate the postgres pod so Velero runs the checkpoint just before it triggers the snapshot.

Snapshotting itself is near instant, the pause on the database is only as long as the CHECKPOINT, which on this estate was under 2 seconds. What is slow is Kopia moving 430 GB of snapshot data to the bucket, and you watch that through the DataUpload objects, not the backup phase, because the backup shows InProgress the entire time the data mover is working.

Restore, bind, and verify the data

On the target, restore from the backup the shared bucket has already synced. Velero recreates the webapp namespace, the PVC data-postgres-0 with its class rewritten to vks-vsan-default by the ConfigMap, and a DataDownload that pulls the 430 GB back out of the bucket and lands it on a freshly provisioned volume. Watch the DataDownload the same way you watched the upload.

Pending, and the class still reads tkgi-vsan-default. That is the ConfigMap not being read, and the cause is almost always a label typo or the ConfigMap living in the wrong namespace. Confirm the labels are exactly velero.io/plugin-config with an empty value and velero.io/change-storage-class set to RestoreItemAction, in the velero namespace, then delete the failed restore and run it again. With the map correct, the PVC binds to vks-vsan-default and the DataDownload starts moving bytes.

That PANIC is the whole reason this part exists. On the first run I moved the volume without the pre-backup CHECKPOINT hook, trusting crash consistency, and the snapshot caught the control file and the log out of step. With the hook in place, the snapshot lands on a clean restart point, PostgreSQL replays 14 seconds of write-ahead log on start, and comes up healthy. Verify data, not liveness, because a pod reporting Ready tells you the process started, not that the rows are all there.

What green looks like: PVC Bound on vks-vsan-default, postgres-0 Ready after a short WAL replay, and the row count on the target matching the source to the row. Match the counts on your two or three busiest tables, not just one, because a partial restore can leave one table short while another looks fine.

flowchart LR
  A[postgres on TKGI] --> B[pre hook CHECKPOINT]
  B --> C[CSI snapshot, one instant]
  C --> D[Kopia upload to bucket]
  D --> E[(shared S3 bucket)]
  E --> F[restore on VKS]
  F --> G[PVC mapped to vks class]
  G --> H[Kopia download to new volume]
  H --> I[postgres starts, WAL replay]
  I --> J[row counts match source]

Rollback and common failures

Rollback stays cheap for the same reason it did last part: you never touched the source. The TKGI database kept serving reads and writes through the entire snapshot and move, because a CSI snapshot does not pause the volume beyond the checkpoint. Backing out means deleting the target namespace and the restore, and the source carries on as though nothing happened.

One caution specific to volumes: if a previous attempt left a PersistentVolume with a Retain reclaim policy, deleting the namespace does not delete that PV, and the next restore can trip on the orphan. List PVs on the target after a rollback and remove any Released volume tied to data-postgres-0 before you retry.

Table 2. Stateful restore failures and their remediation. Rows one and two hit almost every database on the first pass, and row three is the TKGI-specific one people miss.

Field note and verdict

Move the database with a checkpoint, not hope

A clean result looks like this: the change-storage-class ConfigMap applied on the target, the webapp backup taken on tkgi-prod with a pre-backup CHECKPOINT hook and –snapshot-move-data, the DataUpload Completed, the restore run on dev-tkg-01, the PVC data-postgres-0 Bound on vks-vsan-default, postgres-0 Ready after a short WAL replay, and the row counts on your busiest tables matching the source exactly, all while the TKGI database keeps serving. Hit that and you have moved the hard one, and everything left in the wave is a variation on it.

On your own estate on Monday, pick your least critical database, write the storage class map from your two get sc outputs, add a pre-backup hook that quiesces it in one command, and move it with –snapshot-move-data into a fresh namespace on VKS. Do not touch DNS or stop the source. When the PVC hangs, you will already know it is the class map, and when the database will not start, you will already know it is the missing checkpoint. For the volume mechanics underneath this, the Storage and Data Assessment part maps every source volume to its target class. A full migration map lives on the TKGI to VKS guide, and the related VKS, VCF 9 and NSX series sit on the guides hub. Next part moves ingress, load balancing and DNS, the cutover that finally sends traffic to VKS.

Questions worth answering

Why not just use FSB, since it ignores storage class differences?
FSB copies files from the live filesystem with no shared instant, which is fine for static content and fatal for a database whose control file, heap and log must be captured together. Snapshot data movement pins one instant with a CSI snapshot, then moves that. Use FSB only for volumes that have no snapshot support at all.

Is a CSI snapshot not already consistent?
It is crash consistent, meaning it looks like a power cut, which PostgreSQL can usually recover from by replaying its log. Usually is not good enough for production, so the CHECKPOINT hook forces a clean restart point and removes the gamble. It costs under 2 seconds of write pause.

Do the row counts have to match exactly during this rehearsal?
They match to the snapshot instant. If the source keeps taking writes after the snapshot, the target will be slightly behind, and that is expected. Those counts only need to match exactly on the final cutover snapshot in Part 20, when writes to the source are stopped.

Can I move a 500 GB volume faster than 94 minutes?
Bucket throughput and node-agent parallelism set the ceiling, not PostgreSQL. Tune parallel-files-upload and the node-agent concurrency, and put the bucket close to both clusters. Snapshot and replay are already seconds, so there is nothing to win there.

This series covers a production migration. Run the move in a change window against your own environment, quiesce the database with a pre-backup hook, verify row counts against the live source, and leave the source authoritative until the cutover part.

References

• CSI Snapshot Data Movement, Velero documentation
• File System Backup and the TKGI node-agent hostPath, Velero documentation
• Backup Hooks, pre and post commands, Velero documentation

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts