, ,

Storage and Data Assessment for TKGI to VKS, vSphere CSI and Persistent Volumes (TKGI to VKS Series, Part 8)

A PVC that reads Bound is not proof the data moved. This part assesses every persistent volume in a TKGI estate by its backing First Class Disk and storage policy, then maps each one to a VKS target and a migration method that fits its size and downtime budget.

TKGI to VKS Series · Part 8 of 26

A PVC that reads Bound is not proof the data moved. On the pilot I watched a restored claim go green in a VKS cluster while the 40 GiB First Class Disk it depended on still sat in the source cluster datastore, untouched. That bind was real. Yet the data was somewhere else. Storage is the one assessment in this migration where the Kubernetes objects will lie to you, because a PersistentVolume is a pointer and the thing it points at lives in vSphere, not in the cluster.

Key takeaways: Inventory volumes by their backing First Class Disk and storage policy, not by StorageClass name, because the names do not survive the move. VKS provisions through a paravirtual CSI driver that reads storage policies propagated from the vSphere Namespace, while TKGI provisioned through its own CSI install against datastores you named by hand. Three methods exist per volume, Velero data copy, CSI snapshot clone, and static First Class Disk relink, and you pick per volume by size, access mode and downtime budget. reclaimPolicy Delete is the most expensive default in this whole exercise, so set Retain on every production volume before you touch it.
Who this is for: A platform engineer or storage-minded operator holding a TKGI 1.18 estate on NSX-T and planning the VKS target on VCF 9. Terms on first use: a PersistentVolume (PV) is a cluster object that points at real storage; a PersistentVolumeClaim (PVC) is a workload request that binds to a PV; CSI is the Container Storage Interface, the plugin standard Kubernetes uses to talk to storage; a First Class Disk (FCD) is a vSphere managed virtual disk with its own identity, the actual bytes behind a PV; SPBM is Storage Policy Based Management, the vSphere way of tagging datastores with a policy a StorageClass can request; pvCSI is the paravirtual CSI driver that runs inside a VKS guest cluster; CNS is Cloud Native Storage, the vCenter service that tracks Kubernetes volumes as FCDs.

Storage inventory you actually need

Most people start a storage assessment by listing StorageClasses. That is the wrong first column. A StorageClass name is local to a cluster and it will not exist on the far side, so an inventory keyed on it tells you nothing about what has to move. What moves is the FCD, the managed disk in vSphere, and the record of that disk lives in the CSI volumeHandle on each PersistentVolume. Key your inventory on the volumeHandle and everything else, capacity, access mode, reclaim policy, becomes an attribute of a real object you can track from source datastore to target.

Run this against each source cluster and keep the output. For every PV you want the claim it serves, its size, access mode, reclaim policy, the StorageClass it came from, and the volumeHandle that names the FCD behind it. That last field is the join key for the rest of the migration.

# tested against TKGI 1.18, out-of-tree vSphere CSI driver 3.1, target VCF 9.0 VKS, kubectl 1.30 kubectl get pv -o custom-columns="NAME:.metadata.name,CLAIM:.spec.claimRef.name,SIZE:.spec.capacity.storage,MODE:.spec.accessModes,RECLAIM:.spec.persistentVolumeReclaimPolicy,SC:.spec.storageClassName,HANDLE:.spec.csi.volumeHandle" # expected output, prod cluster NAME CLAIM SIZE MODE RECLAIM SC HANDLE pvc-a1e5 postgres-data-0 40Gi [ReadWriteOnce] Delete postgres-fast e5f0c2b1-…-fcd pvc-b2c7 web-config 1Gi [ReadWriteOnce] Delete vsan-default a71c9d33-…-fcd pvc-c9f2 shared-assets 20Gi [ReadWriteMany] Retain shared-nfs 7b40e118-…-fcd

Now resolve each volumeHandle to its disk in vCenter so you know where the bytes physically live and how big they really are on the datastore. CNS is the service that keeps that mapping, and govc reads it without a UI trip.

# read the CNS volume behind a handle, export GOVC_URL and GOVC_PASSWORD from env, never inline the password govc volume.ls -l | grep e5f0c2b1 # expected e5f0c2b1-…-fcd postgres-data-0 40.0GB vsanDatastore attached prod-md-0-xxxx

Across the reference estate, three clusters, that pass returns a short list: one 40 GiB PostgreSQL volume, a couple of 1 GiB config volumes for the web tier, and one 20 GiB ReadWriteMany share. Small in count, but the single 40 GiB volume is where the entire cutover risk sits, and the inventory is what tells you that before you plan anything.

Record three more things while you have the list open, because each one becomes a decision later. First, note any PV whose reclaimPolicy already reads Retain, since those are volumes a careless delete cannot destroy and they are safe to stage early. Second, flag orphaned FCDs, disks that show in CNS with no bound PVC, because a TKGI estate that has run for a few years always carries a handful left behind by failed restores or deleted namespaces, and you do not want to drag dead storage into VCF 9. Third, capture the last known backup time per volume from whatever tool protects it today, so that if a cutover goes sideways you know exactly how much data a restore would cost. A volume with a fresh backup and Retain set is one you can move aggressively, and a volume with neither is one you slow down and protect first.

How TKGI and VKS provision volumes differently

On TKGI you installed and owned the storage stack. You deployed the out-of-tree vSphere CSI driver into each cluster, and your StorageClass referenced either a datastore URL or an SPBM policy you had defined by hand. Your control point was inside the cluster, which is why every cluster could name its classes differently and often did.

VKS moves that control point up into the Supervisor. You assign one or more storage policies to a vSphere Namespace, and vSphere creates a matching StorageClass in that namespace and propagates it into any VKS guest cluster on the namespace. Inside the guest, pvCSI does not talk to vCenter directly; it forwards to the Supervisor CNS-CSI, which owns the CNS conversation. In practice you stop naming StorageClasses and start choosing storage policies, and the class names your workloads see are handed down rather than authored. That is the remap Part 7 warned about at the end, and it is why a lift and shift of PVC manifests fails on the StorageClass field first.

flowchart TB
  subgraph s1 [TKGI storage path]
    T1[StorageClass names a datastore or SPBM policy] --> T2[In cluster vSphere CSI driver]
    T2 --> T3[vCenter CNS creates FCD on datastore]
  end
  subgraph s2 [VKS storage path]
    V1[Storage policy on vSphere Namespace] --> V2[StorageClass propagated to guest]
    V2 --> V3[pvCSI in VKS guest cluster]
    V3 --> V4[Supervisor CNS CSI]
    V4 --> V5[vCenter CNS creates or relinks FCD]
  end
Provisioning control moves from inside the cluster on TKGI to the vSphere Namespace on VKS.

Volume mapping from TKGI StorageClass to VKS policy

Here is the artifact to keep from this part, the volume mapping sheet. One row per source StorageClass, resolved to the target storage policy, the access mode it carries, and the migration method you will use. Build this once and it drives every hands-on storage step later in the series. The reference estate fills in like this.

Source StorageClass (TKGI)Backing store and policyTarget StorageClass (VKS namespace policy)Access modeMethod
postgres-fastvSAN, dedicated performance policyvsan-postgresRWOStatic FCD relink
vsan-defaultvSAN default policyvsan-default (propagated)RWOVelero data copy
web-configvSAN default policyvsan-default (propagated)RWOVelero data copy
shared-nfsNFS datastore, RWX sharevsan-file-rwx (vSAN File Service)RWXRebuild on vSAN File Service

Two rows deserve a note. RWX on TKGI often came from an external NFS datastore, and on VCF 9 the native path is vSAN File Service, which backs ReadWriteMany volumes as vSAN file shares. That is a rebuild, not a relink, because the storage substrate changes underneath. And the postgres-fast row maps to a policy you must create on the target namespace to match the source performance tier, otherwise you will silently land a production database on default storage and discover it under load.

One more column earns its place on that sheet once you start executing, a status field per volume, moved or pending, updated as each wave lands. A migration of even a handful of stateful volumes runs across days, not one maintenance window, and without a running status you will lose track of which disk is authoritative. I keep the sheet in version control next to the manifests, so the mapping and the YAML that acts on it never drift apart, and a teammate picking up the cutover at 2 am can read the current state in one glance instead of guessing.

Migration methods weighed per volume

Every tutorial reaches for the same move: let Velero recreate the PVC on restore and copy the data in with its file level backup. For small config volumes that is fine. For a 40 GiB single-writer database it is the slow and risky path, because restore time scales with data and you are copying bytes that never had to move. When the source and target share a vCenter and can reach the same datastore, a static relink of the existing FCD moves zero data and cuts over in minutes. That is the piece the happy-path guides leave out.

MethodBest forTypical downtimeData copiedMain risk
Velero data copysmall to mid RWO, cross vCenterminutes to hours by sizefull volumerestore time grows with data
CSI snapshot clonesame vCenter, point in time copyshortchanged blockssnapshot must quiesce or you clone crash consistent state
Static FCD relinklarge single writer, same datastore reachableshortestnone, same diskwrong volumeHandle, and the reclaimPolicy trap

Static relink is the method that fails loudly when you get a detail wrong, and the first detail is the StorageClass name. Recreate a PV on the target and forget that VKS renamed the class, and the bound PVC never appears while the pod sits in Pending. Its event log is blunt about it.

kubectl describe pvc postgres-data-0 Events: Warning ProvisioningFailed storageclass.storage.k8s.io "postgres-fast" not found # fix: the target class is vsan-postgres, set spec.storageClassName to the propagated name on both PV and PVC
Field note: On the staging rehearsal I cleaned up a source PVC after a Velero restore reported success. The source PV carried reclaimPolicy Delete, the default TKGI handed us. CNS took that literally and deleted the underlying FCD, the same disk a not yet cutover read replica still mounted. Postgres on that replica went read only inside a minute, and I lost 40 minutes restoring a 40 GiB volume from the last snapshot to get staging back. One field, Retain instead of Delete, set before any cleanup, would have saved the afternoon.

Capacity, access modes and data gravity

Volume size is the variable that decides your method, because it sets how long the app is down. On the pilot I measured the same 40 GiB PostgreSQL volume through all three methods. Velero copied it at roughly 35 MB per second and the cutover ran 19 minutes end to end. A CSI snapshot clone in the same vCenter landed at about 7 minutes. The static FCD relink, which moves no data at all, cut over in 4 minutes and most of that was Postgres restarting cleanly. Those numbers are specific to one estate, but the shape holds: the bigger the volume, the more a copy costs you, and the more data gravity pushes you toward relink.

Cutover downtime for a 40 GiB PostgreSQL volumeMeasured on the pilot, minutes end to end, lower is better0102019 minVelero copy7 minSnapshot clone4 minFCD relink
Same volume, three methods. Copy time is the cost you avoid by relinking an existing disk.

Access mode is the other axis, and it is less forgiving than size. A ReadWriteOnce volume is a single-writer block disk and every method above applies to it. ReadWriteMany is a shared file system, and on VCF 9 that means vSAN File Service, which you enable and configure on the Supervisor before any RWX claim can bind. Do not assume a namespace can serve RWX just because it serves RWO; that is a separate service you turn on, and forgetting it leaves shared-storage apps stuck at deploy time.

Gotcha: A snapshot taken while the database is writing is crash consistent, not application consistent. For Postgres, quiesce with a checkpoint or a brief pause before the snapshot, or accept that recovery on the clone will replay the write ahead log. Test the clone boots clean before you trust it in a cutover plan.

Data gravity deserves a plain rule of thumb, because it is what keeps a migration plan honest. Copy based methods stay comfortable up to a point and then punish you, so anything a single writer owns above roughly 20 GiB is a candidate for relink rather than copy, and anything above 100 GiB should almost never be copied when the disk can simply be reattached in the same vCenter. Weigh two more costs that the size number hides. A copy consumes network and datastore bandwidth that your still running production shares, so a large restore during business hours competes with live traffic. A copy also doubles your storage footprint for the duration, since source disk and target disk both exist until you retire the old one, which matters when vSAN capacity is already tight before you have added a second platform beside the first. Relink avoids both, at the price of a tighter procedure and no second copy to fall back on, which is exactly why you set Retain and take a snapshot before you attempt it.

If you want the component mechanics of how VKS presents storage classes and namespaces, the vSphere Kubernetes Service Complete Guide covers the target platform in depth, and this series links to it rather than re-teaching it. For where this storage work sits in the overall plan, the migration guide holds the full sequence.

What to do with your PVC inventory on Monday

Run the volumeHandle inventory against all three clusters first, because you cannot plan a storage migration you have not resolved down to real disks. Then set reclaimPolicy Retain on every production PV before anything else touches them, so a cleanup mistake cannot delete a live disk. Fill in the volume mapping sheet, one row per source StorageClass, and classify each volume to a method: default to static FCD relink for anything over roughly 20 GiB that a single writer owns and that stays in the same vCenter, use Velero copy for the small config volumes, and rebuild RWX shares on vSAN File Service after you enable it. That short table, built now against your own estate, is what turns the stateful cutover later in this series from a guess into a checklist. Next part stands up Velero on both sides and starts moving the stateless tier, where the same inventory discipline pays off with far less at stake.

TKGI to VKS Series · Part 8 of 26
« Previous: Part 7  |  Guide  |  Next: Part 9 »

References

About The Author


Discover more from Journal of Intelligent Infrastructure

Subscribe to get the latest posts sent to your email.

Leave a Reply

Your email address will not be published. Required fields are marked *

Architect’s Toolkit

About the Author

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

Discover more from Journal of Intelligent Infrastructure

Subscribe now to keep reading and get access to the full archive.

Continue reading