Halfway through the second migration wave I pulled a Velero backup tarball out of the object bucket to check what it had actually captured, ran it through gunzip, and read the production database password out of a base64 blob in a JSON file that four people across two teams had read access to. Nothing leaked and nothing broke, but that single line of output rewrote how I move credential material between platforms, and it is why this Part opens by telling you not to migrate your secrets at all.

Preflight, Secret Inventory and Version Baseline

Last Part we shifted live traffic for the web tier from an NSX T virtual server onto an OpenShift Route and left DNS alone until the old pool went idle. Requests now land on OpenShift. What still binds those pods back to the old platform is everything they read rather than everything they serve: a database password, a registry robot token, an LDAP bind credential, a ConfigMap full of NSX specific hostnames, and a Jenkins job whose only copy of its own credentials lives in a Groovy script on a virtual machine you are about to delete.

Start by finding out how many secrets you actually own, which is almost never the number the API returns. On the reference estate, namespace prod reported 47 secrets. Twelve of them were mine. Everything else was platform bookkeeping that has no business crossing to the new cluster, and knowing that split before you plan the work is the difference between a two hour job and a two day one.

Run the discovery on the source cluster first. Note that on the TKGI side I still reach for kubectl, because it is what the cluster ships with. On the OpenShift side I use oc throughout, and I will say why once: oc is kubectl plus the OpenShift API groups, so commands like oc secrets link and oc adm policy have no kubectl equivalent, and mixing the two clients in a runbook is how a step silently does nothing.

Step 4 is the one people skip and then regret. Two of my twelve real secrets, smtp-relay and feature-flags-token, were referenced by nothing at all. They were left over from a service retired eighteen months earlier and nobody had dared delete them. Migration is a decent excuse to find out that a credential has no consumer, and the correct handling for an orphan is to leave it on the dying platform.

Moving Secrets Without Shipping Them Through Object Storage

Standard migration advice says back the secrets up with the rest of the namespace and restore them on the target. Velero will happily do it, OADP will happily restore it, and every walkthrough you find online does exactly that. It is the wrong call, for two reasons that have nothing to do with whether it works.

First, a Kubernetes secret is base64, not encryption, so a backup that includes secrets writes every production credential you own into an object bucket in a form anyone with bucket access can read in one command. That bucket almost certainly has a wider access list than the namespace did. Second, and more usefully, you are already recreating every consumer of those credentials. Rotation normally costs a change window, a coordination call and a rollback plan. During a platform migration it costs nothing extra, because the rollout that picks up the new value is a rollout you were doing anyway. Skipping rotation here means carrying credentials of unknown age onto a platform you just built clean.

Sort every secret into one of six classes and decide once per class rather than once per object. This table is the artifact worth keeping out of this Part, because four of the six rows resolve to leave it behind, and that is the finding that shrinks the work.

flowchart TD
  S[Secret found on TKGI] --> A{Generated by the platform}
  A -->|yes| L[Leave it behind]
  A -->|no| C{Any workload consumes it}
  C -->|no| L
  C -->|yes| V{Value exists in a vault today}
  V -->|yes| E[Create an ExternalSecret on OpenShift]
  V -->|no| R{Can it be reissued this window}
  R -->|yes| N[Rotate now, create fresh on OpenShift]
  R -->|no| H[Hand create once, schedule rotation inside 30 days]

Step 1. Take the migration backup with credentials excluded

That last label is the part I would not skip. A resource carrying velero.io/exclude-from-backup=true is dropped even when it matches a backup label selector, so it survives someone running a well meaning full cluster backup at 23:00 on cutover eve. Policy that lives on the object beats policy that lives in a runbook nobody reads.

Step 2. Recreate credentials on OpenShift from a system of record

Two mechanisms, and the choice is about who owns the value long term. For anything already in a vault, install External Secrets Operator for Red Hat OpenShift, which is the Red Hat supported build of the external-secrets project and runs cluster wide, fetching and refreshing values into ordinary Kubernetes secrets. If you already had the community operator running, uninstall it first, otherwise the two controllers fight over the same custom resources. For the handful of values with no vault entry yet, create them by hand from an environment variable and never from a file on disk, then put a vault ticket behind them.

Registry credentials deserve their own step because they fail loudly and late. On TKGI most estates end up with one Harbor robot account that every namespace shares, because that was the path of least resistance in 2021. Recreating that arrangement on OpenShift wastes the cleanest opportunity you will get to narrow it. Issue one robot per OpenShift project, and remember that a pull secret does nothing until it is linked to the service account the pods run as. Here is what forgetting that looks like, which is the failure I hit on my first namespace.

Two details in there cost me time. Escaping the dollar sign in the Harbor robot name matters, because an unescaped robot$shop-prod in a double quoted shell expands to robot and the registry rejects it with the same unhelpful unauthorized message. And oc secrets link takes –for=pull for image pulls but –for=mount when a build needs the credential inside the pod, so a build service account usually needs both.

ConfigMaps, Environment Drift and Encryption at Rest

ConfigMaps look like the easy half and they are not, because unlike a secret a ConfigMap usually restores cleanly and then quietly describes the wrong world. On the reference estate, three of nine ConfigMaps in prod contained values that were valid on TKGI and meaningless on OpenShift: an ingress class name referencing NCP, a metrics scrape target pointing at a Telegraf endpoint that only existed on the TKGI worker, and a base URL built from the NSX T virtual server hostname. None of them caused a pod to fail. All three caused something downstream to go quiet, which is much harder to notice.

So diff them rather than trusting the restore. Pull the source ConfigMap, pull the restored one, and read every value that contains a hostname, an IP, a class name or a path. Anything matching your old network deserves a human decision. Below is the diff loop I use, and it caught all three drifted values in under a minute.

While you are in here, deal with encryption at rest, because a fresh OpenShift cluster does not encrypt etcd by default and every credential you just created is sitting in it in the clear. Enabling it is a single patch on the APIServer resource, and OpenShift then encrypts secrets, ConfigMaps, Routes and both OAuth token types. Values only, not keys, so resource types, namespace names and object names stay readable. Do this before the migration wave, not after, because the rollout is measured in tens of minutes and you do not want it running while you are cutting traffic.

Pipeline Migration, Jenkins on TKGI to OpenShift Pipelines

Most TKGI estates build images one of two ways: a Jenkins controller running as a workload on the cluster with dynamic agent pods, or a BuildConfig using the Jenkins pipeline strategy. Both have a problem on OpenShift 4. Pipeline build strategy in a BuildConfig is deprecated, with the replacement being Red Hat OpenShift Pipelines, the supported build of Tekton, a Kubernetes native CI engine where each step is a container and each run is a custom resource. Red Hat publishes a migration chapter for exactly this move. Meanwhile the newer Builds for Red Hat OpenShift, based on the Shipwright project, has split out into its own release train and its own documentation set, so BuildConfig is no longer the single obvious answer for image building either.

My recommendation for a migration, and I have argued myself out of it twice: do not rewrite your pipelines during the cutover. Keep Jenkins running, point its push credentials and its deploy step at OpenShift, get the platform migration finished, then convert to Tekton as a separate piece of work with its own change window. Rewriting CI and moving a platform at the same time means every failed build has two plausible causes and you will spend the difference debugging.

When you do convert, one gotcha will eat your first afternoon. ClusterTask, the cluster scoped task type every Tekton tutorial written before 2025 uses, has been removed from OpenShift Pipelines, and the hub clustertask command went with it. Task resolution now goes through Artifact Hub rather than the retired Tekton Hub, and version pins must be full semantic versions. A pin of 0.9 that worked last year now resolves to nothing.

Numbers from the reference estate, median of nine runs of the same Dockerfile: the Jenkins job on a TKGI agent pod took 400 seconds end to end, of which 95 seconds was waiting for the agent pod to schedule. A first Tekton PipelineRun with a cold workspace took 320 seconds. Once the workspace was a persistent volume claim on the vSphere CSI StorageClass rather than an emptyDir, the same run settled at 145 seconds. Most of that gain is not Tekton being clever, it is the git clone and dependency cache surviving between runs, which the Jenkins setup had also been doing badly. Convert if you want the maintenance story, not because you expect the speed.

Verification, Rollback and Common Failures

Verification here is not is the pod running, because a pod with a stale credential runs perfectly until the moment it does not. Prove three things: every referenced secret exists, every mounted value matches the vault, and no workload still reaches the old platform.

Rollback is genuinely easy at this stage and that is worth stating, because Part 19 and Part 20 were not. Nothing in this Part destroys anything on TKGI. Old secrets stay where they are, the Jenkins controller keeps running, and the old ConfigMaps are untouched. To back out a namespace, scale the OpenShift deployments to zero and shift the load balancer weight back. Rotation is the only irreversible step, so sequence it deliberately: rotate a credential only after the OpenShift workload consuming it is healthy, and never rotate one that a still live TKGI pod is also using, or you will take down the platform you were keeping as your fallback.

Field Note on a Pipeline That Kept Pushing to a Dead Registry

Nine days after we cut the wave two services over, a product owner asked why a bug fix merged the previous Tuesday still was not live. Builds were green. Deployments were green. Every dashboard said the pipeline had shipped eleven times.

What had happened is that we had migrated the pull side and forgotten the push side. Jenkins was still building fine, still authenticating fine, and still pushing to the Harbor project we had marked read only during transition. Harbor accepted the push because the robot still had write permission on the old project. OpenShift, meanwhile, was pulling from the new mirrored project, where the newest tag was nine days stale. Every stage reported success because every stage genuinely succeeded. Nothing anywhere compared the digest that was pushed with the digest that was running.

Cost was about six hours across three people, most of it spent doubting the deployment rather than the build, plus a fairly uncomfortable conversation about a bug fix that had been sitting in a registry nobody was reading. Fix took four minutes: a new push robot scoped to the new project and a one line change to the Jenkins credential binding. The check that would have caught it on day one now runs after every deploy, comparing the image digest the pipeline pushed against the digest the running pod reports.

Verdict on this Part: rotate application credentials and registry robots at the boundary and never carry them through a backup. Copy TLS certificates only when the cutover is same day and reissue is scheduled. And keep Jenkins alive through the migration, converting to OpenShift Pipelines afterwards, because a CI rewrite during a platform move buys you nothing and doubles the number of things that can be blamed.

Rotate at the Boundary, Not After

A clean result for this Part looks like this, and it is worth checking against literally rather than by feel.

Two questions that come up every time. Can you skip External Secrets Operator and just keep creating secrets by hand? Yes, for a small estate, and for twelve credentials it is defensible. It stops being defensible around the point where you have more namespaces than you have people who remember what is in them, because a hand created secret has no owner and no expiry and nothing to remind you it exists. Second question, should any of this push you toward VMware vSphere Kubernetes Service instead, given it keeps more of your existing shape? Possibly, and if credential and pipeline rework is the piece your team cannot absorb this year, read the TKGI to VKS guide before you commit, because that landing place asks less of your CI. It also asks you to stay on the VMware stack, which is the trade.

On Monday, run step 1 of the inventory against one production namespace on your own estate and write down two numbers: how many secrets the API returns, and how many of those any workload actually consumes. If the gap is anything like mine, you have just made this Part four times smaller than you thought it was. Part 22 takes a full non production cluster end to end as a pilot, using everything from Parts 17 through 21. Related ground is covered in Part 15 on OAuth and registry setup and Part 17 on the Velero and OADP toolchain, with the full map in the TKGI to OpenShift guide.

References

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts