A token that lasts ten hours is the reason more pipeline migrations slip than any storage problem. Every time the kubectl vsphere plugin logs in to a VKS cluster it mints a JSON Web Token, a JWT, that expires in roughly ten hours. Copy that kubeconfig into a build runner and the first job after midnight meets a blunt 401. Secrets and config feel like the gentle wave of a TKGI to VKS move, and the plain ones genuinely are, yet the identity your pipeline carries and the sealed material committed to your GitOps repo are exactly where a tidy cutover turns messy.

Where the migration stands now

Last part the web tier answered on its new ingress and DNS pointed at the VKS load balancer. Traffic is flowing, but two things still live only on TKGI: the secrets and config the apps read at start, and the pipeline that builds and deploys them. This part moves both. TKGI is Tanzu Kubernetes Grid Integrated, the outgoing platform, and VKS is where these workloads now run.

Order matters here. Move config and secrets first so a redeploy on VKS finds everything it needs, then repoint the pipeline so the next commit lands on VKS instead of the old cluster. Doing it the other way round gives you a pipeline that deploys into a namespace missing half its secrets, and pods that crash loop on a key that is not there.

Run this per namespace, not per cluster. Each namespace in the running example, webtier and payments and the shared platform namespace, carries its own secrets, config and pull credentials, and folding them into one giant restore makes a failure hard to read. I move one namespace end to end, verify it, then repeat. A wave that touches three namespaces becomes three small, boring restores instead of one large one you cannot reason about when it turns yellow.

Preflight and prerequisites

Before anything moves, prove three things: you can reach both clusters, Velero is installed on the target pointing at the same object store the source backups landed in, and the SealedSecrets controller exists on VKS if you use sealed material. A preflight that skips the object store check is how you discover at restore time that the target Velero is looking at an empty bucket.

Inventory secrets and config on the source clusters

You cannot migrate what you have not counted. List every ConfigMap and Secret per namespace on each source cluster and separate the three kinds that behave differently: plain Opaque objects, sealed objects, and the service account tokens Kubernetes manages for you. Only the first kind is safe to copy verbatim.

Here is the mapping I keep taped to the migration runbook, and the one artifact from this part worth returning to. Read the type, then the method. It is the difference between a restore that just works and a controller log full of decrypt errors.

Two categories catch people out during the count. Config that looks static often carries environment specifics, a database host or a feature flag that differs between the TKGI cluster and VKS, so copy the object and then edit the values that are cluster bound. TLS secrets are the other one: if cert-manager issued them, let cert-manager reissue on VKS against the same DNS name rather than carrying a certificate that is about to rotate anyway.

Migrating ConfigMaps and Secrets with Velero

Velero copies Kubernetes API objects between clusters. Filtered to configmaps and secrets it moves your plain material in seconds. One habit saves confusion: expect a PartiallyFailed status on a first restore into a namespace that already holds service account tokens. Velero skips objects that already exist and warns rather than overwriting, so a stale token from the old cluster produces a warning that reads alarming and means nothing for your app.

If your namespaces are named differently on VKS, Velero maps them with –namespace-mappings old:new on the restore, so a source namespace webtier can land as webtier-prod without editing a single manifest. To silence the stale token warning for good, back up with a label selector that matches only your application objects, since the default service account tokens carry no app labels and simply fall outside the backup.

Sealed secrets, external secrets and the re-seal problem

Here is where the tutorial default is wrong. Guides tell you Velero backs up the whole namespace, so your secrets come along for free. For a SealedSecret that is technically true and operationally useless. A SealedSecret is a Secret encrypted with the public half of a key pair that lives inside one cluster. Copy the encrypted blob to VKS and the controller there, holding a different private key, cannot read it. You get one line in the controller log, no key could decrypt secret, and the plaintext Secret never materialises.

If you already run External Secrets Operator, ESO, the picture is easier. ESO does not store secret values in the cluster or in git, it fetches them from an external store such as Vault at runtime. Migration then means pointing a SecretStore on VKS at the same backend, not copying anything. For teams choosing now, that property is the argument for ESO over sealing: nothing cluster bound to re-encrypt when the next platform arrives.

At twelve sealed objects you do not re-seal by hand twelve times. Loop the live Secrets through kubeseal with the VKS cert, write each sealed file into the GitOps repo, and let your delivery tool apply them, so the committed blobs are the new ones and the old cluster bound blobs stop being the source of truth. Treat the re-seal as a repo change, reviewed and merged, not a one off kubectl apply that nobody can reproduce later.

Pipeline identity, tokens and kubeconfig

A build runner needs an identity that outlives a coffee break. Pinniped, the authentication component the Supervisor wires up for federated login, is built for humans at a keyboard, and the JWT a kubectl vsphere login hands back expires in about ten hours. That is fine for you at a terminal and wrong for a runner that deploys at 2am. Give the pipeline its own ServiceAccount, an SA, with a bound token scoped to the target namespace, and store that token in the CI secret store, never in git.

Bound tokens have a ceiling. A token minted with kubectl create token honours the cluster maximum, often 24 or 48 hours, so a job that runs longer than the token lives will still fail midway. For anything long running, prefer a token your CI platform projects and rotates, or a short token minted fresh at the start of each job rather than one cached for days. Name the kubeconfig context for the cluster and namespace it targets, because a runner with three contexts and the wrong current-context is its own quiet outage.

flowchart LR
  A[Git repo manifests] --> B[CI build stage]
  B --> C[Push image to VKS Harbor]
  C --> D[kubectl apply]
  D --> E[ServiceAccount token auth]
  E --> F[VKS namespace]

One more thing the pipeline carries is the image reference. Your manifests point at the old Harbor, harbor.tkgi.corp.local, and the robot account that pulled from it does not exist on the new registry. Harbor is the container registry both platforms use. Rewrite the reference and recreate the pull secret, or every pod lands in ImagePullBackOff.

Verification, rollback and common failures

Green looks like this. Every app Secret and ConfigMap present in the target namespace with matching keys, the SealedSecrets controller materialising each sealed object with no decrypt error in its log, a pipeline run that authenticates with its ServiceAccount token and pushes to the new Harbor, and pods reporting Running rather than ImagePullBackOff. Prove the secret restore with kubectl get secret db-credentials -n webtier -o jsonpath='{.data}' and confirm the key count matches the source. Prove the pipeline identity by running one job end to end and watching it deploy without a human login anywhere in the logs.

Rollback is first class in a migration. Because you deleted nothing on TKGI, backing out is a pipeline and DNS decision, not a data recovery. Point the pipeline context back at the TKGI cluster, revert the image reference to the old Harbor, and if you had shifted traffic move DNS back. Nothing you did on VKS touched the source, which is the whole reason we migrate beside the old platform rather than in place.

Field notes and a recommendation for pipeline migration

On the production cutover a Jenkins agent had cached an admin kubeconfig from my own kubectl vsphere login. Deploys ran clean all afternoon across all three namespaces. At 2am the nightly job failed, every stage, with You must be logged in to the server. I burned 40 minutes chasing RBAC and network before the timestamp on the token gave it away: it had expired at the ten hour mark, quietly, mid pipeline. Swapping to a bound ServiceAccount token ended it, and I have not shipped a pipeline on a human login since.

For pipeline identity, pick a dedicated ServiceAccount with a bound token, or a projected token if your CI platform speaks OIDC to the Supervisor. Avoid reusing a human kubectl vsphere login for anything automated. For secrets, if you already run a secret manager, pick ESO and repoint the SecretStore. If you run SealedSecrets, re-seal against the VKS controller and avoid dragging the old key across. Copying sealed blobs and hoping is the option to avoid, because it fails silently until a pod cannot start.

Two questions come up every time. Do I need to migrate service account tokens? No. Kubernetes issues new ones on VKS, and copying the old ones only causes the skip warnings you saw. Recreate any token a tool depends on. Can I keep the old Harbor during transition? For a short window yes, if VKS can reach it and a pull secret exists on the target, but cut to the new registry before you decommission TKGI, or you will forget it is still load bearing.

For deeper background, the Velero migration toolchain part covers the backup mechanics this part builds on, the TKGI to VKS guide holds the full map, and the series hub links the related VKS and VCF 9 work.

On Monday, list the SealedSecrets in one source namespace with kubectl get sealedsecrets and re-seal them against your VKS controller. That single dry run tells you whether your GitOps repo is ready for the cutover.

References

Broadcom TechDocs, Understanding Authorization in vSphere Supervisor
Velero docs, Resource filtering for backup and restore
Pinniped docs, Using Pinniped for CI/CD cluster operations

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts