The first stateless service I moved to VKS restored in 22 seconds and then sat there with zero pods running. Its Deployment was healthy, the ReplicaSet existed, and the events log read forbidden: violates PodSecurity restricted:latest three times, once per replica. Nothing was wrong with the app. Everything was wrong with my assumption that stateless means it just moves.

Where the migration stands

Last part we installed Velero on the source TKGI cluster and on the target VKS cluster called dev-tkg-01, pointed both at one S3 bucket, and proved a throwaway namespace could back up on TKGI and restore on VKS. This part carries the first real workload across that pipe, a stateless one, on purpose, because a stateless service has no persistent volume to move and so it isolates every problem that is about the platform rather than the data.

Here is the running example as it stands. The source is a TKGI 1.18 estate on NSX-T with three clusters, dev, staging and prod, a Harbor registry at harbor.platform.local, and one stateful app plus two stateless services. The stateful app, a web and api tier in front of PostgreSQL with a 500 GB volume, waits for the next part. Today moves storefront, a stateless catalog frontend: a Deployment named catalog-web at 3 replicas, a ConfigMap of runtime settings, an image pull Secret for Harbor, and a Service of type LoadBalancer. No database, no PVC, no local disk state. If storefront runs on VKS, the platform is ready for the harder move.

One reframe before any command. People treat a stateless migration as the easy warm-up and reach for backup then restore then done. On VKS that skips the two admission gates the source never had, Pod Security and a pull secret bound to the old registry, and both of them fail after the restore reports success, which is the worst time to find them. So the plan is restore, then clear admission, then verify, then leave the source running.

Preflight on source and target

Three checks earn the restore. You can reach the source with kubectl and log in to the target VKS cluster with the vSphere plugin. The storefront namespace on the source holds only what you think it holds, no stray PVC hiding under a label. And the target can pull the storefront image from wherever it lives, because a stateless migration still needs the container image, and that image sits in Harbor on the estate you are leaving. Prove all three now.

What green looks like: the login prints a context you can select, the PVC query returns No resources found, the object list shows one Deployment, one ConfigMap, the pull Secret and the Service, and the target reports the storefront namespace is not found yet. That last line matters, because Velero by default will not overwrite existing resources, so a namespace already present on the target quietly changes what the restore does.

Note the Service is LoadBalancer with an external IP of 203.0.113.40 handed out by the NSX-T load balancer on the source. That IP belongs to the old estate. On the target, Avi (NSX Advanced Load Balancer) will assign a fresh one, and DNS still points at the old address until Part 20, which is exactly why this move causes no outage.

Back up the stateless namespace on TKGI

Back up the whole storefront namespace from the source. No –snapshot-move-data here, that flag drives volume data movement and there is no volume, so leaving it off keeps the backup to Kubernetes objects only and finishes in seconds. Scope it tight with –include-namespaces so the backup carries storefront and nothing else.

What green looks like: Phase Completed, items backed up equal to items to be backed up, and Backup Volumes reading none, which confirms this really was an object-only capture. That backup took 18 seconds against this namespace. Within a minute the target Velero, which shares the bucket, sees the backup without anything running on the target, the same cross-cluster visibility we proved last part.

Restore onto the VKS cluster

Restore into the same namespace name on the target. Velero recreates the namespace, the Deployment, the ConfigMap, the Secret and the Service. It does not blindly copy the Service network identity: by default Velero strips the assigned clusterIP and lets the target allocate a new one, and it drops the auto-assigned nodePort so the target picks its own. That default is correct for a cross-cluster move and you want it, because the old clusterIP and nodePort belong to the source cluster address space.

Twenty two seconds, and PartiallyFailed rather than Completed. The restore created every object, so the warning is not about Velero, it is about what happens next when the target admission controllers get a look at the pods. This is where a stateless migration on VKS stops being trivial.

flowchart LR
  A[storefront on TKGI] --> B[velero backup, objects only]
  B --> C[(S3 bucket)]
  C --> D[velero restore on VKS]
  D --> E{Pods admitted}
  E -->|rejected| F[Pod Security restricted]
  E -->|imagepullbackoff| G[Harbor pull secret]
  F --> H[label namespace baseline]
  G --> I[recreate pull secret]
  H --> J[catalog-web Ready on VKS]
  I --> J

Pod Security and image pull failures

Look at the pods and there are none, even though the Deployment says it wants three. The ReplicaSet is trying and failing, and its events carry the real reason. Pod Security Admission, the built-in Kubernetes controller that enforces the Pod Security Standards, runs on VKS clusters and enforces the restricted level from cluster version 1.26. TKGI never turned it on, so catalog-web shipped without runAsNonRoot, without a seccompProfile, and without dropping Linux capabilities, and on the source that was fine. On the target it is a hard rejection.

Within a few seconds the ReplicaSet controller retries and the pods schedule. That label is a bridge, not the destination. Baseline lets the workload run while you keep the security posture that made VKS worth moving to, and the real fix is to add a compliant securityContext to catalog-web so it passes at restricted. Do that in the manifest, in your Git repo, not by hand on the cluster, because the next part in the wave will need the same treatment and you want it codified.

Pods schedule and then a second failure lands, ImagePullBackOff. The restored harbor-pull Secret carried a token that the source cluster minted, and the target cannot use it as is, or the Deployment references the image at a Harbor path the target service account cannot authenticate to. A restored pull Secret is the classic thing you should recreate rather than trust. Delete the restored one and create a fresh dockerconfigjson Secret on the target with credentials read from the environment, never inline.

Table 1. What changes when a stateless workload lands on VKS. This is the reference artifact, pin it, because every stateless service in the wave gets processed row by row.

Verify the service serves traffic

Once admission clears and the pull secret is fixed, the three replicas go Ready and the Service picks up an external IP from Avi. Verify the workload from inside the cluster first, before anyone outside knows it exists, because DNS still points at the source. Curl the new external IP directly and confirm the app answers.

What green looks like: three pods Running and Ready, a new external IP of 198.51.100.61 from Avi that is not the old 203.0.113.40, and a 200 from the health endpoint. Time from restore to first healthy pod on this run, once the two fixes were in place, was 47 seconds. Meanwhile the source storefront on TKGI never stopped serving during any of this, so users saw nothing.

Rollback and fallback

Rollback here is the easiest in the whole series, because you never touched the source. Your TKGI storefront kept running and serving its old IP the entire time, so backing out means deleting what you created on the target and walking away. Nothing about the source cluster, the DNS records or the users changed.

Because the cutover has not happened, this is a true fallback with zero user impact, and it stays that way right up until Part 20 moves DNS. Until then you can restore, break, delete and restore again as many times as you like. Treat that freedom as a gift and use it to get the manifest right.

Table 2. Stateless restore failures and their remediation. Rows one and two hit almost every estate on the first pass.

Field note and verdict

Move the smallest stateless service first

A clean result looks like this: storefront backed up on tkgi-prod as an object-only backup, restored onto dev-tkg-01, the namespace labelled so Pod Security admits the pods, a freshly created Harbor pull secret, three catalog-web replicas Running and Ready, a new Avi external IP recorded, a 200 from the health check, and the source storefront still live on its old IP with DNS untouched. Hit that and you have proven the platform carries a real workload, not just a throwaway namespace.

On your own estate on Monday, pick the least important stateless service you run, back it up with –include-namespaces and no snapshot flag, and restore it onto your first VKS cluster into a fresh namespace. Do not touch DNS. When the restore goes PartiallyFailed, you will already know to check Pod Security first and the pull secret second. For the registry side of that pull secret, the Identity, RBAC and Harbor on VKS part sets up the target registry properly. The full migration map lives on the TKGI to VKS guide, and the related VKS, VCF 9 and NSX series sit on the guides hub if you need the component mechanics. Next part moves the hard one, a stateful application with a 500 GB persistent volume.

Questions worth answering

Why did the restore say PartiallyFailed if every object restored?
Velero restored the Kubernetes objects successfully, but the target admission controllers rejected the pods the Deployment tried to create afterward. That rejection surfaces as a warning on the restore, not as a Velero error, which is why the log hunt is a dead end.

Should I keep the namespace at baseline for good?
No. Baseline is a bridge so the workload runs while you fix the manifest. Add a compliant securityContext, then move the namespace back to enforce=restricted so it matches the posture the rest of the VKS estate holds.

Do I have to recreate every restored Secret?
Not every Secret, but every credential Secret that authenticates to something outside the cluster, pull secrets and registry tokens especially. Config-only Secrets restore fine. Anything holding a token minted by the source is suspect on the target.

Can users reach the app on the new IP already?
They can if you point them at the Avi IP directly, but do not. DNS still resolves to the source, and cutting traffic over is a deliberate step in Part 20 with its own validation and rollback. Verify internally now, cut over later.

This series covers a production migration. Run the move in a change window against your own environment, verify on the new IP internally, and leave the source serving real traffic until the cutover part.

References

• Configure Pod Security Admission for VKS Clusters, Broadcom TechDocs
• Restore Reference, Service and nodePort behavior, Velero documentation
• Pod Security Admission, Kubernetes documentation

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts