Our first volume inventory across the three TKGI clusters came back with 47 PersistentVolumes and 3.4 TB provisioned. Twenty two of those volumes were Released, or bound to a namespace somebody had deleted months earlier, so a third of what looked like migration scope was data nobody had claimed in over a year.
Discovery Commands for a TKGI Volume Inventory
Everything in this part was checked against a specific set of versions. Storage behaviour drifts between minor releases more than most people expect, so pin what you are testing before you trust any output.
One note on tooling before the commands. On the TKGI side I use kubectl, because that is all TKGI ships. On the OpenShift side I use oc throughout the rest of this series, because oc carries the project, route and SCC subcommands that kubectl has no concept of, and mixing the two in a runbook is how people end up filing bugs against the wrong binary. I will say that once and then stop mentioning it.
Start with a single wide listing of every PersistentVolume, because the default kubectl output hides the two columns that matter most, reclaim policy and the namespace that owns the claim.
A trimmed sample of what came back on the production cluster:
Two rows in that sample are already telling you something. A Released phase means the claim is gone but the volume is still sitting on a datastore burning capacity, and a claimRef with no namespace means even the pointer back to an owner has been garbage collected. Neither belongs in a migration wave. Both belong on a decommission list you hand to the vSphere team later.
Next, separate the volumes by what actually provisioned them. TKGI estates that have been running for a few years usually carry a mix, some volumes created by the older in tree vSphere Cloud Provider and some by the vSphere CSI driver that TKGI installs into its clusters. Those two populations behave very differently when you try to move them.
Look at what the in tree rows contain. A datastore bracket, a folder UUID and a VMDK filename. That path is meaningful only to the vCenter and datastore the TKGI cluster was attached to. Nothing in it survives a move to a different cluster with its own CNS records, and upstream Kubernetes finished retiring the built in cloud provider integrations in the 1.31 line anyway, so the object shape you might have hoped to lift and drop is on its way out regardless. Treat in tree volumes as data with a filesystem in it, not as objects to be transplanted.
Finally, get the real number rather than the provisioned one. Provisioned capacity is what your finance model uses, but used capacity is what sets your restore window, and on our estate the gap was almost three to one.
For used capacity you have to ask inside the pod, because Kubernetes does not track filesystem consumption on a PV. A short loop over the running pods with df gives you a number you can plan against, and it is worth the ten minutes it takes.
Access Modes and ReadWriteMany Exposure
Access mode is where a storage assessment either finishes quietly or turns into a procurement conversation. ReadWriteOnce claims, which is most of what a TKGI estate carries, land on OpenShift without argument. Each one becomes a block volume on the vSphere CSI driver, attached to a single node at a time, exactly as it was.
ReadWriteMany is different, and it is different in a way that has a hard dependency outside your cluster. Red Hat is explicit about it in the OpenShift storage documentation: RWX provisioning through the vSphere CSI Driver Operator works only where the underlying vSphere environment has vSAN file services configured. Without that, a request for RWX does not degrade gracefully. Volume creation fails and an error is logged, and your PVC sits in Pending until somebody notices.
Two claims out of forty seven. That is a typical ratio and it is exactly why RWX gets missed. It looks like a rounding error in a spreadsheet, right up until wave one stalls on it. Here is what the failure looks like on the OpenShift side when the file backend is not there:
You have three ways out, and the right one depends almost entirely on how much RWX you actually have. Enabling vSAN file services keeps everything inside the vSphere CSI driver you already run, and costs you a vSAN configuration change plus the file service appliances. OpenShift Data Foundation gives you CephFS and a much richer storage platform, and costs you a subscription and a real operational learning curve. An existing NFS export, wired in statically or through the CIFS and SMB CSI Driver Operator, costs you almost nothing and gives you almost nothing beyond a shared directory.
One more RWX detail that bites later rather than sooner. Resizing an RWX vSphere volume requires detaching the disk from every instance using it, which on a shared assets volume means stopping every pod that mounts it. Plan capacity generously on RWX at migration time, because growing it afterwards is an outage, not an operation.
Storage Class Mapping from TKGI to OpenShift
Here is the artifact worth keeping from this part. Print it, stick it in your migration wiki, and fill the last column in with your own names. Every row below is a decision somebody has to make before the first stateful workload moves, and every row has at least one way to get it wrong quietly.
| On TKGI | OpenShift 4 equivalent | Migration action | Where it goes wrong |
|---|---|---|---|
| In tree volumes, provisioner kubernetes.io/vsphere-volume | thin-csi, provisioner csi.vsphere.vmware.com | Move the data, let OpenShift create a fresh PV | volumePath points at a datastore the OCP cluster may not even mount |
| Custom StorageClass with a storage policy name | Custom StorageClass with StoragePolicyName parameter | Recreate by hand, do not restore the TKGI object | The SPBM policy must be visible to the vCenter and datastore OpenShift uses |
| ReadWriteOnce PVC on vSAN or VMFS | thin-csi block volume, ReadWriteOnce | Data moved by OADP file system backup or the CSI data mover | thin-csi ships with allowVolumeExpansion set to false |
| ReadWriteMany PVC on NFS or vSAN file share | vSAN file services RWX, ODF CephFS, or an NFS or SMB CSI class | Choose the file backend before wave one is scheduled | No backend means PVC Pending, not a fallback to RWO |
| Reclaim policy Retain on production data | Set reclaimPolicy Retain on your own StorageClass | Create a retained class alongside the default | thin-csi defaults to Delete, so a namespace delete takes the data |
| CNS volume snapshots taken on TKGI | VolumeSnapshotClass on the OpenShift cluster | Nothing, snapshots are not migration transport | vSphere CSI allows 3 snapshots per volume by default, 32 at most |
| StatefulSet volumeClaimTemplates | Identical StatefulSet with a rewritten class name | Restore the whole StatefulSet, never single PVCs | The class name is immutable inside the template once created |
Now the failure that catches almost everyone on their first restore, because it produces no error from Velero at all. Your TKGI PVCs name a class called fast-vsan. OpenShift has thin-csi. Velero happily recreates the PVC exactly as it was backed up, the API server accepts it, and then nothing happens forever.
Velero has a built in restore item action for precisely this, driven by a labelled ConfigMap rather than a flag. Create it once, before your first restore, and every PVC and PV in every future restore gets rewritten on the way in.
Restore Windows and Data Movement Sizing
Here is where I disagree with almost every migration plan I have reviewed. Teams reach for snapshots, because snapshots are instant and every storage vendor has trained them to think that way. On a TKGI to OpenShift move, snapshots are the one transport that cannot work. A vSphere CNS snapshot belongs to the source cluster and its CNS records. It is not an object another cluster can consume as a data source, and no amount of RBAC makes it one.
What actually moves data is a copy through object storage, either file system backup with Kopia or the CSI snapshot data mover, which snapshots locally and then uploads the snapshot content. Both write to a bucket and read it back on the far side. That makes restore duration a function of sustained throughput and concurrency, not of how fast your all flash vSAN can clone a VMDK. Storage teams consistently underestimate this by an order of magnitude.
Read the red line carefully, because it explains a class of failure that looks like corruption and is not. Red Hat documents the DataMover default resourceTimeout as 10 minutes, and recommends raising it to 60 minutes for a 500 GB volume. At the throughput we measured, anything above roughly 60 GB of used data blows through the default. What you see is a restore that reports failure while the data is still flowing, and an engineer who assumes the backup is bad.
Keep these defaults in front of you when you size a window.
| Setting | Default | What to plan for |
|---|---|---|
| DataMover resourceTimeout | 10 minutes | 60 minutes for a 500 GB volume, per Red Hat guidance |
| Velero file system backup restore timeout | 4 hours | Raise with –fs-backup-timeout above about 1.5 TB per restore |
| Kopia node agent resources | Unbounded | 20 cores and 32 Gi handled over 100 GB in Red Hat testing, set limits |
| thin-csi allowVolumeExpansion | false | Create your own class with expansion on before wave one |
| thin-csi volumeBindingMode | WaitForFirstConsumer | A restored PVC stays Pending until a pod schedules, which is correct |
| vSphere CSI snapshots per volume | 3 | Up to 32, though 2 or 3 performs best |
That WaitForFirstConsumer row deserves a sentence of its own, because it changes what a healthy restore looks like. On TKGI with immediate binding, a restored PVC binds straight away and you know within seconds whether storage worked. On OpenShift with thin-csi, a freshly restored PVC sitting in Pending with no events is the expected state until the workload is scheduled. Half the storage escalations I saw during our pilot were people looking at a perfectly healthy Pending PVC.
Pod admission interacts with all of this too. A restored database pod that wrote as root on TKGI will be rejected by the default restricted-v2 Security Context Constraint on OpenShift, and the volume it never mounted will look like a storage problem in your incident channel. I covered that mechanism in detail in Part 7 on Security Context Constraints, and the fsGroup behaviour there is worth rereading before you move any stateful workload.
Field Note from a 47 Volume Assessment
We ran this assessment across dev, staging and production over four days. Provisioned capacity came to 3.4 TB. Used capacity, once we had walked every running pod with df, came to 1.1 TB. Twenty two of the 47 volumes were Released or orphaned, which took another 640 GB out of scope before we had moved a single byte. That reframed the whole conversation with the infrastructure team, who had budgeted a migration window sized on the 3.4 TB number.
Then we found the RWX claim. An 80 GB shared assets volume, mounted by four replicas of the web tier, holding customer uploaded images. Nobody had flagged it in the application assessment because from the developer side it was just a directory. We found it six days before the pilot cutover, and enabling vSAN file services on the target datastore needed a change window that did not exist inside those six days. Pilot slipped nine days.
I had originally put OpenShift Data Foundation in the target design specifically to cover RWX. Once we knew it was one claim of 80 GB, that decision looked ridiculous. We reversed it, mounted an existing NFS export through a static PV, and saved a subscription line plus every hour of operational learning that comes with running Ceph. If we had run the RWX query in week one rather than week seven, the nine day slip would have been a Tuesday afternoon.
Worth saying plainly, because a reader in this position has a genuine fork in front of them. If your organisation is committed to staying on the VMware stack and VCF 9 is on your roadmap, VMware vSphere Kubernetes Service is a different landing place with a different storage story, and I cover it separately in the TKGI to VKS guide. Everything from here on assumes you have already made the OpenShift call.
Drop Orphaned Volumes Before Sizing Waves
My recommendation for this part is narrow and it is the one thing I would insist on. Do not size a single migration window until you have separated bound volumes from orphaned ones and used capacity from provisioned capacity. On our estate those two corrections cut the data movement problem from 3.4 TB to 1.1 TB, and then to under 500 GB once the orphans were gone. That is the difference between a weekend cutover and a three week programme, and it costs you an afternoon of kubectl.
After that, run the RWX query on every cluster before you finalise the target architecture, because it is the only storage finding that can add a procurement decision to your critical path. Everything else on the list, class renames, expansion flags, reclaim policies and timeouts, is configuration you can fix in an hour once you know about it.
A clean result for this part looks like four artifacts on your wiki: a per cluster volume inventory with phase and used size, a decommission list of orphans with owners named, a completed storage class mapping table, and a signed off decision on the RWX backend. If any of those four is missing when wave one starts, storage will be what delays you.
On Monday, run the two jq queries in this part against your busiest TKGI cluster and count how many PersistentVolumes come back Released. Whatever that number is, it is capacity you are paying for and workload you do not have to migrate. Next part builds the target reference architecture that all of this feeds into, and the volume numbers you collect today are an input to it. Identity and multi tenancy design came earlier in the series guide if you need to catch up on ordering.
References
- Using Container Storage Interface (CSI), Storage, OpenShift Container Platform 4.18, Red Hat Documentation
- Restore Reference, including changing PV and PVC storage classes, Velero 1.18 documentation
- OADP application backup and restore, Backup and restore, OpenShift Container Platform 4.19, Red Hat Documentation
- PersistentVolume Storage Options on vSphere, Tanzu Kubernetes Grid Integrated Edition 1.19, Broadcom TechDocs
About The Author
DrJha