1. Run the pilot on the OpenShift cluster you intend to keep, not on a throwaway sandbox. Most of what breaks is cluster configuration, and a sandbox carries different configuration.
2. Write the exit criteria before anything moves. Mine ran to 14 lines and every line is a command with a pass condition. A criterion that cannot be expressed as a command is an opinion.
3. Headline command:
velero backup create pilot-staging-20260816 --include-namespaces wave1-web,wave2-data --snapshot-volumes=false --default-volumes-to-fs-backup --wait4. Hand the keyboard to the application team. A pilot executed by the platform engineer holding cluster-admin proves only that the platform works for the one person who already knows the answers.
5. Budget double. My six hour plan took 9 hours and 40 minutes, and every minute of overrun landed in identity and verification, not in data movement.
A pilot that passes tells you almost nothing. A pilot that fails in a way you can name, on a cluster that matters, in front of the team that will inherit the runbook, tells you everything worth knowing before production day. That distinction drives every choice in this Part, starting with a deliberate decision about where to go and break something.
Pilot Scope, and Why Staging Not Dev
Our reference estate carries three TKGI (Tanzu Kubernetes Grid Integrated) clusters: dev, staging and prod. Instinct points at dev, because nobody files a ticket when dev falls over. That instinct costs you the pilot. Dev has no LDAP groups anyone cares about, certificates are self signed, the database holds a few gigabytes of seed rows, no dashboard is watched, and no team is inconvenienced by an outage. Each of those absences quietly removes a failure mode you specifically need to trigger while the stakes are low.
Staging is different in exactly the ways that matter. It shares the production LDAP group structure, so identity is real. Its certificates come from the corporate certificate authority, so trust chains are real. Its database is restored weekly from a production dump, so data volume and row counts are real. And an application team watches it, so a broken deployment produces a complaint within twenty minutes. That last property is not a risk to be managed, it is the instrument you are trying to read.
Staging holds 34 namespaces. Two of them move in the pilot: wave1-web, the stateless tier from Part 18, and wave2-data, the PostgreSQL tier from Part 19. Between them they exercise every mechanism in the series: a Deployment behind an Ingress, persistent volumes with an image that wants a fixed user identifier, secrets sourced from a vault, network policy between tiers, and a Jenkins pipeline that pushes to Harbor. Tutorials tell you to pilot with your simplest workload. Do the opposite. Your simplest workload will restore cleanly and teach you nothing, and you will discover the admission and identity walls in production instead.
flowchart TD
S[Scope two namespaces on staging] --> X[Write exit criteria as commands]
X --> P[Preflight both platforms]
P --> B[Backup on TKGI with Velero]
B --> R[Restore on OpenShift with OADP]
R --> V{All criteria green}
V -->|no| F[Fix cluster config, roll back, log it]
F --> P
V -->|yes| Z[Freeze the runbook and the timings]
Z --> C[Production cutover]
Exit Criteria Scorecard, Written Before Anything Moves
A pilot without written exit criteria always passes, because the person running it decides at the end what passing meant. Write the list first, circulate it, and let the application owner add two lines of their own. My version ran to 14 rows and it is the artifact from this Part worth keeping. Each row names a criterion, the command that proves it, and the output that counts as a pass. Anything you cannot reduce to a command belongs in a risk log, not in the scorecard.
| # | Criterion | Proving command | Pass condition |
|---|---|---|---|
| 1 | Object storage reachable from both clusters | velero backup-location get | Phase Available on source and target |
| 2 | Backup completes without errors | velero backup describe $VELERO_BACKUP | Phase Completed, Errors 0 |
| 3 | Target projects exist and are Active | oc get project wave1-web wave2-data | Both Active |
| 4 | Object counts match the source | oc get all -n wave1-web –no-headers | wc -l | Equals the source count taken at freeze |
| 5 | Every workload pod is Running | oc get pods -n wave2-data | No CrashLoopBackOff, no Init errors |
| 6 | No SCC escalation was needed | oc get pod -o jsonpath of openshift.io/scc | restricted-v2 or nonroot-v2, never anyuid |
| 7 | Persistent volume claims are bound | oc get pvc -n wave2-data | All Bound, capacity matches source |
| 8 | Data landed intact, not merely present | psql -tAc select count from orders | Row count equals the frozen source count |
| 9 | Route serves the application | curl -sSI https://shop.pilot.example.com | HTTP 200 within 2 seconds |
| 10 | Certificate comes from the corporate CA | openssl s_client -connect host:443 | Issuer is the corporate CA, not the default ingress certificate |
| 11 | Application team can work without cluster-admin | oc login with an LDAP account, then oc get pods | Succeeds with project scoped rights only |
| 12 | Network policy still isolates the tiers | oc rsh into a web pod, curl the database service | Refused unless an explicit policy allows it |
| 13 | Pipeline builds and deploys to the target | tkn pipelinerun list -n wave1-web | Succeeded, image lands in the internal registry |
| 14 | Rollback was executed and timed | stopwatch, from decision to source healthy | Under 15 minutes, TKGI serving 100 percent |
On my first run, 11 of those 14 came back green. Rows 6, 11 and 14 failed, and all three failures were platform configuration rather than application behaviour. That ratio is normal and it is the reason the pilot exists. If you score 14 out of 14 on a first attempt, check whether you ran every step yourself as a cluster administrator, because that is the usual explanation.
Preflight and Version Baseline
Pin versions in writing before the first command, because a pilot whose result you cannot reproduce three weeks later is a story rather than evidence. On the OpenShift side use oc rather than kubectl, since only oc knows about projects, routes and SCC review commands. TKGI has no oc, so the source side stays on kubectl throughout.
Preflight then measures the pilot rather than describing it. Count the objects and the bytes, because both numbers become verification inputs later and both feed the timing estimate. No credential appears in any manifest below. Every access key is read from an environment variable that the shell sources from the vault, which is the pattern established in Part 21.
That failure had nothing to do with the key itself. Our DataProtectionApplication referenced a secret whose data key was named credentials, while the AWS style plugin looks for a key named cloud. Six days of a healthy looking Operator install hid it, because nothing validates a backup storage location until something asks it to. Add row 1 of the scorecard for exactly this reason, and run it the morning of the pilot rather than the week before.
Running the Pilot, Hour by Hour
Step 1, Freeze Writes and Back Up on TKGI
Scale the web tier to zero first, capture the row count that verification will compare against, then take the backup. Capturing the count after the freeze rather than before is what turns row 8 of the scorecard from a guess into a proof.
Step 2, Prepare Projects and Identity on OpenShift
Here is where my first run came apart, and where a sandbox cluster would have lied to me. Following the reference architecture from Part 11 we removed the self provisioner binding, so ordinary users cannot mint projects. That is correct policy and it is also the reason the application team could not start.
Step 3, Restore in Two Passes
Restore the stateless namespace first and let it settle, then restore the data namespace. Two passes cost about eight minutes more than one and buy you a clean attribution when something fails. A single combined restore that reports PartiallyFailed forces you to read a hundred lines of describe output to work out which tier hurt.
Step 4, Point Pilot Traffic at OpenShift
Pilot traffic moves the same way production traffic will, using the weighted pool technique from Part 20, with the staging DNS record dropped to a 60 second time to live the day before. Rehearsing the mechanism matters more than the traffic volume. If your pilot shifts traffic by editing a hosts file, you have rehearsed nothing that production will use.
Step 5, Stop the Clock and Publish the Timings
Record elapsed minutes per phase while you work, not from memory afterwards. These numbers are how you size the production window in Part 23, and they are the only part of the pilot that a director will actually read. My plan said six hours. Reality said 9 hours and 40 minutes, and the shape of the overrun was more instructive than the total.
Verification, Rollback and Common Failures
Green looks like this. Note that the SCC annotation is checked explicitly, because a pod running under anyuid will look perfectly healthy and quietly fail your security review two months later.
Rollback is not a contingency in a pilot, it is a scorecard row you must execute even on a successful run. Nothing on TKGI was deleted, so backing out is a traffic decision followed by a cleanup. Time it with a stopwatch and write the number down.
| Failure seen in the pilot | Cause | Remediation |
|---|---|---|
| BackupStorageLocation Unavailable, InvalidAccessKeyId in the velero log | Credential secret key named credentials, plugin expects cloud | Recreate the secret with the key name cloud, then delete the velero pod to force revalidation |
| You may not request a new project via this API | Self provisioner role removed, group holds no project creation right | Pre create projects with oc adm new-project and bind admin to the LDAP group |
| storageclass fast-vsan not found during restore | TKGI StorageClass name does not exist on OpenShift | Apply a change-storage-class ConfigMap in openshift-adp before rerunning the restore |
| forbidden: unable to validate against any security context constraint | Pod spec carries a fixed runAsUser that restricted-v2 rejects | Remove runAsUser and let the project allocate, or bind the service account to nonroot-v2. Avoid anyuid. |
| Route returns 503 while pods are Running | Service has no ready endpoints because the readiness probe targets a port the Route does not use | oc get endpoints, align the probe port with the Service targetPort |
| Restore PartiallyFailed on a Deployment resource | Manifest still uses an API version the OCP 4.19 server no longer serves | Rewrite to apps/v1 at source before the backup, and add an API deprecation scan to preflight |
| node-agent pods in CrashLoopBackOff on the source cluster | Kopia repository credential secret missing from the velero namespace | Recreate the repository credential and restart the daemonset before retrying the backup |
Field Note on a Pilot That Passed Because I Ran It
On an earlier engagement I ran a pilot exactly like this one and declared it green in five and a half hours. Six namespaces restored, smoke tests passed, route served 200s, database row counts matched. I wrote a confident summary and we scheduled the first production wave for eleven days later.
Every command in that pilot came from my laptop, under a kubeconfig with cluster-admin. When the application team ran the identical runbook with their own LDAP backed accounts, 3 of the 6 namespaces stopped at the first step, because they could not create a project and nobody had noticed that I never needed to. Then a second wall appeared behind the first: their group had edit rather than admin on the projects we did pre create, so they could not bind a service account to nonroot-v2 either, which is the exact operation the stateful tier requires. Two full days went into rewriting the role bindings and rerunning the pilot, and the production wave slipped by a week.
Nothing was technically wrong with the platform. What was wrong was the experiment. I had tested whether OpenShift could host the workloads, when the question that decides a migration is whether the people who own the workloads can move them under the permissions they will actually hold. Since then the rule in my runbooks is blunt: the platform engineer may sit in the room and may not touch the keyboard. If a step cannot be completed by the application team, it is a defect in the platform configuration, and finding it during a pilot costs an hour instead of a week.
Pilot on Staging, With the Application Team Driving
My recommendation for this Part is one sentence: pilot on staging, on the OpenShift cluster you intend to keep, with your two ugliest namespaces, and let the application team hold the keyboard while you take notes. Avoid the dev cluster and avoid a disposable sandbox. Both will hand you a pass you cannot bank.
A clean result looks like this. All 14 scorecard rows green on a rerun, with the first run failures written down and fixed in cluster configuration rather than in the application. Per phase timings recorded, so the production window in the next Part is arithmetic rather than optimism. A rollback actually performed and timed, under 15 minutes. A runbook frozen at a version number, executed at least once by someone who is not you. And a short list of platform defects closed, each with the command that proves the fix.
Two questions come up every time. First, can the pilot namespaces stay on OpenShift afterwards. Yes, and they should, because a pilot you tear down stops generating evidence the moment it ends. Leave staging running on OpenShift and let it accumulate a fortnight of operational history before production day. Second, does a pilot on VMware vSphere behave differently if the target is VKS rather than OpenShift. The method transfers, the admission and identity failures do not, and that comparison belongs in the TKGI to VKS Series rather than here.
On Monday, open a document and write your 14 exit criteria against your own estate before you touch a cluster. If you cannot write the proving command for a row, you have found the first gap in your platform, and you have found it for free. Full context for the whole migration sits in the TKGI to OpenShift guide.
References
- OADP application backup and restore, OpenShift Container Platform 4.19, Red Hat Documentation
- File System Backup, Velero 1.16 documentation
- OpenShift Container Platform Life Cycle Policy, Red Hat Customer Portal
About The Author
DrJha