Six minutes after a restore that reported Completed, my database pod sat in CrashLoopBackOff and its log carried exactly one useful line: FATAL: data directory /var/lib/postgresql/data/pgdata has wrong ownership. Nothing had gone wrong with the backup. Nothing had gone wrong with the restore either.

OpenShift had done exactly what it is built to do, which is refuse to run a container as the user identifier baked into the image. That one line is the entire distance between Part 18 and this Part. Moving a stateless Deployment is a YAML problem. Moving a StatefulSet, meaning a workload whose pods keep stable identities and stable volumes, with 61 GiB of live PostgreSQL underneath it, is four problems stacked on top of each other: YAML, data consistency, storage classes and admission. They surface in that order, and only the first one is easy.

Preflight and Backing Up wave2-data on TKGI

Two conventions before any command runs. On the OpenShift side I use oc rather than kubectl, because it is a superset that understands Projects, Routes and Security Context Constraints, and half of this Part is about the last of those. On the TKGI side there is no oc, so kubectl it is. Mixing them by accident is how people end up debugging the wrong cluster at midnight.

Version pinning matters more here than anywhere else in the series. OADP, which is OpenShift API for Data Protection and is Velero packaged as an Operator, bundles a specific Velero server. OADP 1.5 ships Velero 1.16. Upstream Velero has since moved to 1.18. Installing 1.18 on TKGI because it is current, then asking a 1.16 server to restore what it wrote, is an avoidable class of problem. Pin the source to match. A second reason to care: from Velero 1.17 the restic path is disabled for new backups entirely, so kopia is the only file system uploader left, and you want that decision made deliberately rather than discovered.

Step 1. Point the node agent at the TKGI kubelet path

File system backup, or FSB, is the mode where Velero copies the contents of a mounted volume rather than asking the storage layer for a snapshot. It runs inside a DaemonSet called node-agent, and that DaemonSet mounts the kubelet pod directory from each host. Upstream defaults assume /var/lib/kubelet/pods. TKGI does not put it there. BOSH deployed nodes keep kubelet data under /var/vcap/data, and if you skip this, every pod volume backup fails with a missing path under /host_pods and no obvious reason why.

Step 2. Quiesce Postgres with a pre-backup hook

Velero is explicit that FSB copies data from a live file system, so the bytes are not captured at a single point in time. For a web asset directory that is fine. For a database with an active write path it is a coin toss, and I lost that toss once already, which is the war story further down. Backup hooks exist precisely to close this gap. Annotate the pod so Velero runs a command inside the container before it starts copying. Note the command value has to be a JSON array; a bare shell string silently does nothing useful because Velero does not run it through a shell by default.

Credentials never go in the annotation. PGPASSWORD and the user come from the environment the container already has, sourced from the existing Secret, so the hook reads them and nothing sensitive is written into an annotation that every reader of the namespace can list.

Setting on-error: Fail on the pre-hook is deliberate. If pg_dump cannot run, I want the backup to fail loudly rather than produce a file system copy that looks complete and is not. A post-hook removes the dump afterwards so the source volume does not slowly fill with migration artefacts across repeated dry runs. That dump lands inside the data directory on purpose, because that is the directory FSB is about to copy, which means the logical dump travels with the physical copy in one object and there is no second bucket to reconcile.

Step 3. Run the backup and read what it actually captured

Read HooksAttempted and HooksFailed every single time. A backup can report Completed with a failed hook if you set on-error: Continue, and that is the exact shape of a backup that will restore into a corrupt database. Read the byte counts too. If pgdata came back at a few hundred megabytes when the claim holds 61 GiB, the node agent read an empty host path and reported success on nothing.

StorageClass Mapping and Target Project Preparation

A PersistentVolumeClaim, or PVC, carries the name of the StorageClass it was provisioned from. That name is local to a cluster. My TKGI clusters used a hand written class called tkgi-vsphere-gold. An installer provisioned OpenShift 4 on vSphere creates one called thin-csi. Restore a PVC referencing a class that does not exist and it sits Pending forever with a provisioning event that names the missing class. Nothing about that failure is subtle, but it happens at minute 80 of a cutover window, which is the worst possible time to discover it.

Velero solves this with a restore item action driven by a labelled ConfigMap. Both labels are mandatory and both are easy to typo. It lives in the OADP namespace, which is openshift-adp, not in the application namespace, and Velero reads it at restore time with no restart required.

Step 4. Map the source StorageClass to thin-csi

Step 5. Create the target project and read its UID range

Every OpenShift project gets an allocated block of user identifiers, and admission validates pod security context against it. You want those numbers written down before the restore, because in twenty minutes an error message will quote them back at you and you should recognise them rather than guess.

Restoring Volume Data with OADP

OADP does not install a velero binary on your workstation. It runs one inside the Velero Deployment, and the supported way to drive it is an alias that execs into that container. Restore into a pilot namespace first, never straight over the name you intend to serve production from. A namespace mapping costs nothing and buys you the ability to run this twice.

Step 6. Restore into a pilot namespace

PartiallyFailed is the normal first result and it is not a reason to start over. Velero backs up each resource at whatever API version the source cluster preferred, and restoring needs that same group and version to exist on the target. My TKGI cluster ran the Prometheus Operator, so it had a ServiceMonitor custom resource definition. My OpenShift cluster had user workload monitoring switched off, so it did not. Two valid choices: enable the feature first, or exclude the resource from the restore and add monitoring back once the application is stable. I chose the second, because a cutover window is not the moment to change a cluster wide monitoring setting.

Ownership, fsGroup and Admission

Part 7 covered why Security Context Constraints, or SCCs, are the biggest portability gap in this migration. Here is what that looks like when a real database hits it. A community PostgreSQL image pins itself to a fixed user identifier and sets a matching group. TKGI, with a permissive PodSecurityPolicy posture, never questioned it. OpenShift default policy is restricted-v2, which enforces a run as user range drawn from the project annotation you noted in Step 5, and enforces an fsGroup drawn from the same block. A pinned identifier of 999 is not in a range that starts at 1000670000, so the StatefulSet controller cannot create the pod at all.

Step 7. Fix admission, then fix ownership

That fsGroupChangePolicy line is not decoration. By default OpenShift walks every file on a mounted volume and rewrites ownership to match fsGroup. On a 61 GiB data directory with a few million small files that took just over 6 minutes on first mount, during which the pod is not ready and a readiness probe with a short failure threshold will happily restart it into a loop. Setting OnRootMismatch makes the kubelet check the top level directory and skip the recursive walk when it already matches, which took the same mount to under 20 seconds on every subsequent start.

flowchart TD
  A[Pod rejected at creation] --> B{Image pins a fixed UID}
  B -->|No| C[Stay on restricted v2]
  B -->|Yes| D{Can you rebuild the image}
  D -->|Yes| E[chgrp 0 and chmod g equals u on written paths]
  D -->|No| F{Does it only need non root}
  F -->|Yes| G[Grant nonroot v2 to the service account]
  F -->|No| H[Grant anyuid and file a dated exception]
  E --> I[Set fsGroup plus fsGroupChangePolicy OnRootMismatch]
  C --> I
  G --> I
  H --> I
  I --> J[Verify with the scc annotation on the running pod]

My verdict, and it has not changed across four of these migrations: rebuild the image. Grant nonroot-v2 only as a bridge with a ticket attached, and treat anyuid as something you have to defend to an auditor, because eventually you will. A migration is the one moment when nobody argues about touching a Dockerfile, and if you spend an SCC grant instead, that grant outlives everyone who understood why it was made.

Verification, Rollback and Common Failures

A running pod proves the platform accepted your workload. It proves nothing about your data. Verification for a stateful migration is arithmetic, and it has to be arithmetic taken on both sides within the same quiet window.

Rollback deserves a sentence of its own. Up to and including this Part, nothing on TKGI has been modified. Users are still served by the old cluster, the old NSX-T load balancer and the old DNS record. Backing out is one oc delete project and a conversation you do not have to have. That changes in Part 20 when DNS moves, so extract every ounce of value from cheap rollback while you still have it.

Field Note from a Restore That Was Green and Wrong

On my first real attempt at this namespace there was no pre-backup hook. I did not think one was needed, because the plan was to quiesce the application at the load balancer and I assumed a quiet application meant a quiet data directory. Velero reported Completed. OADP restored into the pilot namespace. Postgres started, the web tier connected, and a colleague ran a smoke test that passed. Everything looked finished at 11:40 on a Saturday.

At 15:20 someone compared row counts because I had asked for it as a formality. TKGI had 2,418,077 rows in the orders table. OpenShift had 2,411,930. A gap of 6,147 rows, all of them written in the eleven minutes the backup was running, because a background reconciliation job I had forgotten about was still inserting while the load balancer sat drained. No error was ever raised, at any layer, by any tool. The only signal that anything was wrong was a number somebody bothered to check.

That cost three days. One to re-plan, one to add hooks and rehearse them across every database in the wave, and one to rebuild trust in a migration plan that had just quietly lost six thousand orders in a lab. I reversed a decision I had defended in design review, which was that file system backup alone is sufficient for a database if you quiesce traffic. It is not. Traffic is not the only writer. Since then every database in every wave gets a logical dump written into the volume by a pre-backup hook, and every migration gets a row count on both sides before anything is called done.

Dump the Database, Then Move the Volume

My recommendation for every stateful workload in this migration is a belt and braces one, and I make no apology for it. Move the volume with file system backup because it lands you a clean, freshly provisioned PersistentVolume on OpenShift with no memory of vSphere objects owned by another cluster. Carry a logical dump inside that same volume because file system backup is honest about its consistency limits and a dump costs you a few gigabytes and twenty minutes. Verify with a count, never with a pod status. Then decide the identity question deliberately: rebuild the image for an arbitrary user identifier if you possibly can, and if you cannot, grant the narrowest SCC to a single service account and write down when that grant expires.

Part 20 takes what is now a working application in a pilot namespace and puts real traffic on it, converting Ingress objects to Routes and moving DNS off the NSX-T load balancer. If a landing on VMware vSphere Kubernetes Service rather than OpenShift is still on your table, the equivalent stateful hop is covered in the TKGI to VKS Complete Guide, and the admission story there is considerably shorter. Otherwise, do this on Monday: pick your smallest production database, run the backup with a hook, restore it into a pilot project, and compare one row count. Whatever breaks will break in miniature, which is the only good time for it.

Related reading in this series: Part 7 on Security Context Constraints, Part 9 on storage assessment, and Part 17 on the toolchain itself. Everything above sits on top of all three, and the full map is in the TKGI to OpenShift Complete Guide.

References

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts