My first cross platform restore reported Phase: Completed with zero errors and zero warnings. Eleven minutes later the same workload on OpenShift had a Deployment sitting at 0 of 3 replicas and a ReplicaSet event reading forbidden: unable to validate against any security context constraint. Both of those things were true at once, and closing that gap is most of what building this toolchain actually involves.

Two Parts back this series settled the question of what tool to use. Red Hat’s Migration Toolkit for Containers, MTC, migrates between OpenShift clusters and from OpenShift 3 to 4, and its source has to be an OpenShift cluster. A TKGI cluster is not one, so MTC has no entry point here at all. What remains is Velero, the upstream backup and restore engine, running twice: once on TKGI where it reads your workloads, and once inside OADP, the OpenShift API for Data Protection, which is Red Hat’s supported packaging of Velero as an Operator. Object storage sits between them and is the only thing the two clusters share.

Preflight, Object Storage and Version Skew

Before installing anything, settle three decisions, because each one is expensive to reverse after you have written a terabyte into a bucket.

First, buckets. Use two, not one. Bucket tkgi-migration holds everything Velero on TKGI writes. Bucket ocp-backups holds everything OADP writes for OpenShift’s own protection. They serve different lifecycles and different owners, and mixing them is how source backups get deleted. More on that below, because it cost me a weekend.

Second, the version pin. Velero maintainers test the upgrade path across n minus 2 minor releases, meaning a build tagged 1.16 is verified against backups produced by 1.15 and 1.14. Backup file format follows semantic versioning, so older minors usually restore fine, but usually is not a support statement. OADP 1.5.0 moved the bundled Velero server from 1.14 to 1.16, and Red Hat supports exactly one OADP version per OpenShift minor, so on OpenShift 4.19 your target side is Velero 1.16 whether you like it or not. That fixes the number you have to hit on the TKGI side.

Third, the data mover. On TKGI you have two choices, file system backup through the node agent, or the Velero plugin for vSphere which takes First Class Disk snapshots. Snapshots are faster to take and useless to you, because a vSphere snapshot handle restores to vSphere volumes owned by the same plugin, and OADP on the target side is not running that plugin. File system backup produces a portable Kopia or restic repository that anything speaking Velero can read. Take the slower option. Portability is the entire point of this exercise.

Client 1.16.2 on one side and server 1.16.0 on the other is a patch difference, which is inside the tested window with room to spare. If your TKGI release only ships a signed Velero 1.11 or 1.12, you are five minors behind the target and outside anything Velero verifies. Two options exist and neither is fun: run an unsigned upstream Velero 1.16 binary on TKGI and accept that the source side is unsupported for the duration of the migration, or restore through an intermediate. I take the first, because the source cluster is being decommissioned anyway and a support case against a platform that reaches End of Support in October 2027 is not the safety net people imagine it is.

That flatness is the useful part. File system restore is bound by file count and object storage round trips, not by link speed, so a bigger pipe buys you very little. Anybody sizing a maintenance window from raw network bandwidth will be wrong by an order of magnitude. Measure one real namespace, divide, and plan from that number.

Installing Velero on a TKGI Cluster

Velero CLI context follows kubectl context, so set that first with tkgi get-credentials and confirm you are pointed at the cluster you think you are. Install with the node agent enabled and volume snapshots disabled, which forces every volume through file system backup.

Three crashing node agents on a fresh install is not a broken download, it is TKGI being TKGI. Velero’s DaemonSet mounts the kubelet pod directory at the upstream path /var/lib/kubelet/pods. BOSH deployed nodes put it under /var/vcap/data/kubelet/pods, so the mount resolves to nothing and the agent exits. Broadcom documents the fix and it is a one line edit.

Raise the memory limit now. Velero ships with a 256Mi limit and a backup that walks tens of thousands of files will sit at status=InProgress for hours without ever printing an error, because the process is being throttled rather than killed. Doubling it to 512Mi took my longest namespace backup from a stall I abandoned after 4 hours down to 51 minutes.

Installing OADP on OpenShift 4

On the OpenShift side you do not run velero install. You install the OADP Operator from OperatorHub into the openshift-adp namespace and then declare a DataProtectionApplication, the custom resource that OADP reconciles into a Velero deployment, a node agent DaemonSet and backup locations. I use oc rather than kubectl throughout the OpenShift half of this series, and will keep doing so, because oc understands Routes, Projects, SCCs and oc adm policy, none of which kubectl knows exist.

Two backup locations go in the same DataProtectionApplication. One points at the TKGI bucket and is marked ReadOnly. One points at the OpenShift bucket and is the default.

First Round Trip, Backup on TKGI and Restore on OpenShift

Pick the smallest stateless namespace you have and run the whole loop once before anything real depends on it. My pilot is wave1-api, four gigabytes, no persistent volumes, seven pods.

flowchart LR
  A[TKGI cluster, wave namespaces] --> B[Velero server and node agent]
  B --> C[Kopia repository in bucket tkgi migration]
  C --> D[BackupStorageLocation tkgi source, ReadOnly]
  D --> E[Velero inside OADP on OpenShift]
  E --> F[Restore into project]
  F --> G[SCC admission decides if pods start]
  E --> H[Bucket ocp backups, writable, OpenShift own backups]

Excluding podsecuritypolicies.policy matters and it is easy to forget. PodSecurityPolicy was removed from Kubernetes in 1.25 and does not exist on OpenShift 4.19, so leaving it in the restore set produces resource errors that clutter the log and hide the failures you actually care about. Your TKGI namespaces are full of PSP bindings that mean nothing on the target. Drop them at the door.

Now for the part that surprises everyone the first time. Phase Completed, three warnings, and no running pods.

Velero did its job perfectly. It wrote a Deployment that asks to run as UID 1001, exactly as it ran on TKGI under a permissive PodSecurityPolicy. OpenShift’s restricted-v2 Security Context Constraint, the admission policy that decides which UIDs, groups and capabilities a pod may request, allocates each project a UID range and rejects anything outside it. Part 7 of this series covered why this gap exists and how to decide between fixing the workload and granting a constraint; the short version is that removing runAsUser and fsGroup from the manifest so OpenShift assigns from the project range is the right fix roughly four times out of five, and granting nonroot-v2 to a dedicated ServiceAccount covers most of the rest.

What matters for the toolchain is the sequencing lesson. Admission runs after restore, on pod creation, not on object creation. Velero cannot see it and will never report it. If you plan a migration wave around Velero exit codes you will declare victory while nothing is serving traffic.

Verification, Rollback and Toolchain Failures

Green looks like four things, checked in this order, and a restore is not done until all four pass.

Rollback is genuinely cheap here and that is the best property of this toolchain. Nothing you have done touches TKGI. Velero on the source cluster only reads; the workloads keep running and keep serving. To back out of a failed restore, delete the OpenShift project and start again. Traffic has not moved, DNS has not moved, and the source of truth is still where it was this morning.

Six failures account for nearly everything I have hit building this pipe on three clusters. Keep this table next to the runbook.

Field Note from a Backup That Reported Success

I built the first version of this pipe with one bucket, because two felt like ceremony. TKGI wrote to it, OADP read from it, and I registered the location on OpenShift the way every tutorial does, writable, default settings, no thought given. That worked for nine days.

On the tenth morning three backups covering two namespaces were gone. Not corrupt, not unreadable. Gone from the bucket, and gone from velero backup get on both clusters. Velero on TKGI had stamped a 720 hour TTL on those backups by default, the clock had not run out, but I had also created a short lived test backup with a 24 hour TTL and OADP’s garbage collector, doing precisely what it is documented to do on a writable location, cleaned up what it considered expired and pruned repository data that the older backups shared. Total cost was one weekend: 5 hours and 8 minutes to re-run the wave2-db backup alone, plus a Monday morning conversation with a database team who had signed off on a migration window that no longer had a valid restore point behind it.

I reversed the design that afternoon. Two buckets, and accessMode: ReadOnly on the source location, which is a single line in the DataProtectionApplication and is the most valuable line in this entire part. Nothing on the OpenShift side has deleted a source backup since. If you take one thing from Part 17, take that line.

A second habit came out of the same week. Set an explicit TTL on every migration backup and make it longer than your entire migration programme, because the default is 30 days and a wave that slips twice will outlive its own restore point. I use –ttl 4380h, six months, on anything that backs a production cutover.

Pin Both Sides to One Velero Minor

My recommendation for this part is narrow and I will defend it: decide your Velero minor version from the OpenShift end, then make TKGI match it. OADP gives you no choice, since Red Hat supports one OADP release per OpenShift minor and that release carries a fixed Velero server. Everything upstream of that is negotiable, so negotiate it. Running an unsigned upstream Velero on a source cluster that reaches End of Support in October 2027 is a smaller risk than restoring across five minors of format skew that nobody has ever tested.

Avoid the Velero plugin for vSphere on the source side for migration work. It is a fine tool for backing up TKGI to restore onto TKGI, and it is a dead end for anything leaving the platform. If you are still weighing whether to leave the platform at all, the TKGI to VKS migration guide covers the landing place where vSphere native snapshots do stay useful, and the storage mapping work behind all of this sits in Part 9 on storage and persistent volumes.

A clean result at the end of this part looks like this. Velero runs on all three TKGI clusters with node agents healthy and memory raised. OADP 1.5.0 runs on OpenShift with two backup locations, one of them ReadOnly. A backup taken on TKGI appears in oc -n openshift-adp get backup within one sync interval without anybody copying a file. One stateless namespace has completed a full round trip, and any SCC rejections it produced are written down rather than worked around by hand. You have a measured restore rate for your own storage, expressed in gigabytes per hour, that you can put in front of an application owner.

Two questions come up every time I run this with a team. Do you need Velero on every TKGI cluster, or one central install? Every cluster, because the node agent has to read pod volumes from the node it runs on, and there is no remote mode. Can OADP restore a backup taken by a newer Velero than it runs? Assume no. Format compatibility is documented in one direction, older backups into newer servers, and the reverse is where people discover that a major format bump is a real thing.

On Monday, install Velero on your least important TKGI cluster, patch the hostPath, and back up a single namespace. You will know within an hour whether your object storage, your credentials and your network path work, and every later wave rests on those three things. Next in this series we take a real stateless application through the whole journey, from waves defined in Part 6 to traffic on OpenShift.

References

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts