How do we know this actually works before we bet production on it? That question came from the release manager in a go/no-go meeting, and a pilot is the only honest answer to it. A pilot migration runs the entire playbook against one non-prod cluster, from standing up VKS through cutover and validation, so the production move repeats something you have already done rather than attempting it for the first time with real users watching.

Pilot goals and where the migration stands

Last part moved secrets, config and the delivery pipeline for the dev workloads onto VKS. Every piece now exists on the new platform, but you have never run the pieces together as one continuous sequence. A pilot does exactly that: pick one non-prod cluster and migrate it end to end, timing each phase, so the run becomes a script you follow in production instead of a set of steps you reassemble under pressure.

Here is where the common pilot advice falls down. Most guides pilot with a stateless hello world app because it goes green fast and looks like success. A stateless demo proves your network path and nothing about your data, and data is where production cutovers actually break. Scope the pilot to include your one real stateful app, the PostgreSQL tier with persistent volumes, or you are rehearsing the easy half and calling it done.

Which non-prod cluster you pick matters as much as the scope. A pristine sandbox with one demo namespace runs clean and teaches you nothing, because production does not look like that. Pick the dev cluster that carries the same auth integration, the same ingress class, and a workload shaped like production, a web tier in front of a database, so the failures you hit in the pilot are the failures you would hit for real. If dev is too quiet, seed it with a copy of a production namespace first, then pilot against that.

flowchart TD
  A[Stand up dev VKS cluster] --> B[Migrate stateless wave]
  B --> C[Migrate stateful wave with volumes]
  C --> D[Cutover and smoke test]
  D --> E{Exit criteria met}
  E -->|Yes| F[Record timings, promote plan to prod]
  E -->|No| G[Rollback, delete VKS cluster, fix, retry]

Preflight for the pilot cluster

Before the pilot starts, prove four things: you can reach the source dev cluster and the target VKS cluster, Velero is installed on both and pointing at the same object store, a vSphere Namespace exists for the pilot with compute and storage quota, and you have a rollback path that touches nothing on TKGI. A preflight that skips the quota check is how a cluster sits in Provisioning for twenty minutes before anyone reads the events.

Two preflight checks earn their place by catching silent failures. Confirm the target Velero and the source Velero name the same bucket and prefix, because a restore against an empty bucket reports success with zero items and looks identical to a restore with nothing to do. Confirm the pilot vSphere Namespace has a storage policy attached and quota above the sum of your persistent volumes, or the first stateful claim provisions into a policy that does not exist and sits Pending with a message nobody reads until the pod will not start.

Standing up the dev VKS cluster

First real step is a cluster to migrate into. Apply a Cluster manifest into the pilot vSphere Namespace and let the Supervisor provision a control plane and workers. In vSphere 9 the kubectl vsphere plugin and the tanzu CLI fold into one VCF CLI, so a login that used to need two tools now needs one, though the kubectl vsphere login command still works for continuity.

Match the Kubernetes version in the manifest to a vSphere Kubernetes release the Supervisor actually offers, or the cluster never provisions. List the available releases with kubectl get tkr before you write the version field, because a version that looks current can be one the content library has not finished syncing. In vSphere 9 the VCF CLI replaces the separate kubectl vsphere and tanzu binaries, so on a fresh workstation install that one plugin rather than chasing two.

My first pilot cluster sat at Provisioning for a quarter of an hour with no worker nodes appearing. Reading the events told the story: the VM class the manifest asked for was not associated with the pilot vSphere Namespace, so the machines had nothing to schedule onto.

Running the migration waves end to end

With a cluster ready, run the same waves you rehearsed in isolation, now back to back. Stateless services go first because they carry no data and validate the network path, then the stateful tier with its persistent volumes. Velero backs up on the source and restores on the target, and with CSI enabled it snapshots the volumes as part of the backup rather than needing a separate step. CSI is the Container Storage Interface, the plugin layer that lets Kubernetes drive vSphere storage.

Order inside the stateful wave is where a pilot earns back its cost. A volume snapshot taken while PostgreSQL is mid write captures a torn state that restores into a database that will not start, so quiesce writes first. In the pilot I scale the web tier to zero for the ninety seconds the snapshot takes, or use a Velero pre-backup hook that runs a checkpoint, then let traffic back. Skipping that step is how a restore comes up with a row count short of the source and a corruption line in the log.

If the pilot namespace names differ from the source, Velero remaps them on restore with the namespace-mappings flag, so a source namespace payments can land as payments-pilot without editing a manifest. Keep the names identical where you can, because every remap is one more thing that is true in the pilot and different in production, and the point of a pilot is to remove those differences, not add them.

First restore of the stateful namespace left the database pod stuck and the whole pilot stalled behind it. A PVC cannot bind to a StorageClass that does not exist on the target, and the TKGI class name almost never matches the VKS one. PVC is a PersistentVolumeClaim, a pod’s request for a volume of a given class and size.

Cutover validation and smoke tests

Green is not the pod reporting Running. Green is the application answering on its new address with its data intact. Smoke test three layers: pods and PVCs bound, the ingress serving real responses, and the database holding the row count you recorded on the source before the backup. A row count that matches within zero is the single check that separates a real data migration from a hopeful one.

A false green is the trap here. Pods report Running because the container started, not because the app is healthy, and an ingress can answer 200 from a default backend that knows nothing about your service. Test the path a user takes, a request that reads from the database and returns real data, not a static health endpoint that a dead backend still serves. In the pilot I hit one real read and one real write, then confirm the write landed, because a read only test misses a broken storage mount that only fails when something tries to persist.

Here is the artifact worth keeping, the pilot exit criteria I score before calling a go or a no-go. Each line is pass or fail, measured not guessed, and a single fail is a no-go until the cause is understood. Copy this table into your runbook and fill the measured column live during the pilot.

Rollback, cleanup and common pilot failures

A pilot you cannot back out of is not a pilot, it is an unplanned production move. Because nothing on TKGI was touched, rolling back is deletion on the VKS side, not data recovery. Delete the Velero restores, delete the pilot cluster, and the source dev cluster is exactly where it started. Clean up the pilot even on a success, because a half deleted pilot cluster is the resource nobody remembers paying for two months later.

Deletion is not the only cleanup that matters. Preserve two things from every pilot even when you tear the cluster down: the phase timings, which become the production change window, and the diff between the runbook you started with and the runbook you finished with, because every fix you applied live is a step missing from the written procedure. A pilot that leaves the runbook unchanged either went suspiciously well or was not watched closely enough.

Verdict on pilot scope and your next move

My first ever TKGI to VKS pilot used a stateless demo app and nothing else. It went green in about twenty minutes and everyone left the room confident. Two weeks later the production attempt hit the real PostgreSQL tier, the PVC would not bind because of the StorageClass name, and I burned ninety minutes on a failure the pilot could have caught for the price of one extra namespace. I have not scoped a pilot without the real stateful app since.

For pilot scope, pick the dev cluster that mirrors production topology and include your hardest workload, the stateful one. Avoid the empty sandbox pilot that migrates a demo and proves nothing about volumes or data. For the run itself, time every phase and keep the numbers, because the sum of those phase timings, plus a margin for the human parts, is the change window you request for production.

One more scoping decision trips teams up, how large to make the pilot cluster. A single worker node runs the demo and hides scheduling problems that only appear when pods spread across nodes and a volume has to attach on the node the pod landed on. Match the pilot worker count to the dev source, three nodes in the running example, so anti-affinity rules and volume attachment behave the way they will in production. A larger pilot costs a few more VMs for an afternoon and buys a real test of placement.

One pilot is rarely enough. A dev run shakes out the mechanical failures, the VM class binding and the storage class map, and a second pilot against staging proves them fixed and rehearses the human coordination, the DNS change and the sign off, that dev never exercises. Two pilots sound expensive until you price ninety minutes of production downtime against an afternoon of rehearsal.

Two questions come up whenever I run this with a team. Do we need a fresh cluster for the pilot, or can we reuse one? Use a fresh cluster you can delete, because a reused cluster carries leftover state that masks the exact provisioning and binding steps production will face. Can the pilot and production share one Velero backup? No, take a fresh backup for each run, since a backup captures a point in time and production data will have moved on from whatever the pilot froze.

For the mechanics this pilot chains together, the Velero toolchain part sets up the backup engine, and the stateful application part covers the volume restore that is the pilot’s real test. The TKGI to VKS guide holds the full map, and the series hub links the related VKS and VCF 9 work.

On Monday, write your pilot exit criteria before you touch a cluster, one row per thing that must be true to call a go. That single page turns your pilot from a demo into a decision, and the next part takes that decision into a production cutover.

References

Broadcom TechDocs, Using the vSphere Kubernetes Service
Velero docs, release schedule and supported versions
Velero on GitHub, backup and migrate Kubernetes applications

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts