Our first pass at the estate counted 34 namespaces. Two weeks later the real number was 47, and the gap was not sloppiness, it was a discovery method that trusted the wrong command as its source of truth.

What Discovery Has to Produce

Discovery on a migration is not an audit. An audit asks whether the platform is healthy. Discovery asks a narrower and more useful question: for each thing running on TKGI today, what has to change before it will start on OpenShift. Those are different outputs, and teams that run an audit first waste a fortnight producing a document nobody uses for planning.

A migration inventory has one job, which is to feed wave planning. Wave planning needs to know which namespaces are cheap to move, which are expensive, and which are blocked until somebody rebuilds a container image. You get that from twelve fields per namespace. Anything beyond twelve fields is a project that never finishes, and anything below eight leaves you guessing at cutover.

Below is the worksheet we run against every cluster. Copy it into a spreadsheet, one row per namespace, and populate it with the commands in the next section. This is the artifact you will come back to for the rest of the migration, and it is the single deliverable of this Part.

Prerequisites and Preflight Checks

You need four credentials before you start, and it pays to prove all four in one go rather than discovering a missing one halfway through a cluster sweep. You need an Ops Manager (Tanzu Operations Manager, the appliance that installs and configures TKGI) login, a BOSH Director login, a TKGI API login whose UAA account carries the pks.clusters.admin scope, and cluster admin on each Kubernetes cluster. If your TKGI account only carries pks.clusters.manage, stop and get the admin scope. That single detail is what produced the 34 versus 47 gap in the opening paragraph.

Versions this Part was written and tested against: TKGI 1.18, BOSH CLI 7.5, Ops Manager CLI (om) 7.x, a kubectl 1.27 client against the TKGI supplied server, and on the target side OCP 4.20 with the matching oc client. Do not assume a TKGI patch level maps to a Kubernetes minor you remember, because it drifts across patches. Read the Product Snapshot in your own release notes and read kubectl version on the running cluster. On the OpenShift side this series prefers oc over kubectl throughout, for one reason worth stating once: oc understands Projects, Routes, ImageStreams and the SCC review subresources that kubectl cannot reach without raw API calls. On the TKGI side kubectl is correct and oc buys you nothing.

If that last check prints no, fix it before going further. A namespace scoped kubeconfig does not fail loudly on most of the commands that follow. It quietly returns a subset, and a subset inventory is worse than no inventory because it looks finished.

flowchart TD
  A[Ops Manager tile] --> B[BOSH Director]
  B --> C[bosh deployments, authoritative list]
  C --> D[Reconcile against tkgi clusters]
  D --> E{Counts match}
  E -- no --> F[Chase orphan or foreign owned clusters]
  E -- yes --> G[Per cluster kubectl sweep]
  F --> G
  G --> H[Workload and image inventory]
  G --> I[Storage and PVC inventory]
  G --> J[Network and Ingress inventory]
  G --> K[Admission risk inventory]
  H --> L[Estate worksheet, one row per namespace]
  I --> L
  J --> L
  K --> L
  L --> M[Wave planning in Part 6]

Estate Discovery, Step by Step

Six steps, roughly 40 minutes of command time on a three cluster estate. Run them in order. Steps 1 and 2 establish scope, and if you skip straight to step 3 you will sweep the wrong set of clusters very thoroughly.

1. Build an authoritative cluster list from BOSH

Every TKGI cluster is a BOSH deployment named service-instance_ followed by its UUID. BOSH does not care who created the cluster or which UAA scopes you hold, so this list is complete by construction. Start here.

Four against three. On our estate the missing deployment was a cluster created two years earlier by a departed engineer under a different UAA account, still running, still consuming 12 vCPU and 48 GB of memory, and still serving an internal reporting tool that a finance team used every month. Nobody in the migration kickoff mentioned it because nobody in the room could see it.

Resolve the delta by asking BOSH for the deployment properties and reading the cluster name out of the manifest.

2. Pull a kubeconfig per cluster into isolated files

Do not sweep four clusters through one kubeconfig with context switching. One stale context and you attribute prod namespaces to dev, which you will not notice until wave planning contradicts itself.

3. Sweep namespaces and workloads

Exclude the system namespaces TKGI owns, because none of them migrate. On TKGI that means kube-system, kube-public, kube-node-lease, pks-system and vmware-system-csi. Everything else is a candidate Project on OpenShift.

Those four DaemonSets deserve a note in the worksheet right now. A DaemonSet on TKGI is usually a log shipper or a monitoring agent mounting /var/log through a hostPath, and on OpenShift the platform already ships that function through cluster Operators. Most of them should be deleted rather than migrated, and deciding that early removes work from every later wave.

4. Inventory storage, then load balancing and Ingress

Storage first, because persistent volumes set the ceiling on how fast a wave can move. Every PVC row you capture becomes a line in the StorageClass mapping table that Part 9 builds and Part 19 executes.

That ReadWriteMany row on nfs-legacy is the one to circle. vSphere CSI on OpenShift does not offer ReadWriteMany block volumes, so either you keep an external NFS provider on the target or the application changes. Finding that in week one is worth more than any other single line in the inventory.

Seventeen of 63 Ingress objects carried NCP specific annotations, and zero NetworkPolicy objects existed anywhere. Both numbers say the same thing. Network behaviour on this estate is not described in Kubernetes manifests, it lives in NSX-T, and nothing you back up with Velero will carry it across. If you want the background on how distributed firewall rules are actually expressed, the NSX Series treatment of DFW and security policy covers it properly and this series will not repeat it.

5. Score admission risk, the field that reorders your waves

Here is where most inventories go wrong. Operators reach for PodSecurityPolicy, because that is what TKGI used to enforce, and get an error they misread as a permissions problem.

Three namespaces at privileged and a pile at unset, which on a cluster with no cluster wide default behaves as permissive. That is the inherited posture the series keeps warning about: TKGI let these pods run, so nobody ever fixed them. OpenShift will not. Its default Security Context Constraint, restricted-v2, drops all Linux capabilities and forces runAsUser into the range allocated to the Project by the openshift.io/sa.scc.uid-range annotation. Leave runAsUser empty and you are fine. Pin it to 0, or to any value outside that range, and the pod needs nonroot-v2 or anyuid, which somebody has to justify and grant.

So count the four patterns that restricted-v2 refuses, per namespace, and put the total in the worksheet.

6. Catch deprecated API usage before the OCP API server does

Kubernetes exposes a counter for exactly this, and almost nobody reads it. It tells you which clients are still calling API versions that a newer server will refuse.

Verification and What Green Looks Like

Discovery is finished when four numbers reconcile, not when the commands stop printing. Check these in order and do not move to wave planning until each one holds.

Green looks like this: 4 clusters, 47 namespaces, 312 running pods, 22 PVCs totalling 1,843 GiB, 9 LoadBalancer Services, 63 Ingress objects, 0 NetworkPolicy objects, 99 pods carrying admission risk, and 6 deprecated API group and version pairs. Nine numbers on one line. If you cannot state your estate in nine numbers, discovery is not done.

Backing Out of a Read Only Sweep

Rollback matters even here, and treating it as unnecessary is how a discovery exercise turns into an incident review. Nothing in this Part writes to a cluster, but three side effects are real and two of them have bitten us.

First, tkgi get-credentials writes credentials to disk. Four kubeconfigs, each with a bearer token valid for the lifetime of your UAA session, sitting in a home directory that is probably backed up somewhere. Clean them up when the sweep is done: shred -u ~/t2os/kubeconfigs/*.yaml. Second, the metrics endpoint in step 6 and the full pod list in step 5 both hit the API server hard on a large cluster. On our prod cluster the kubectl get pods -A -o json call returned 41 MB and pushed apiserver p99 latency from 180 ms to just over 2 seconds for about eleven seconds. Nothing failed, but if you run that during a deployment window somebody will page you. Run sweeps outside change windows and add --chunk-size=200 if the cluster is bigger than ours.

Third, if you did have to add pks.clusters.admin to your UAA account to complete step 1, remove it afterwards. An elevated scope granted for a two hour discovery task and never revoked is exactly the finding an auditor will pick up six months later, and you will not remember why it is there.

Common Discovery Failures and Fixes

Field Note from a Three Cluster Sweep

We planned discovery as a two day task and it took nine working days. Six of those extra days trace to one decision made in the first hour: we opened with tkgi clusters because that is the tool the platform gives you, wrote down three clusters, and built a wave plan around them.

The fourth cluster surfaced in week three, when a finance analyst raised a ticket asking whether the reporting tool would move too. Nobody on the migration team had heard of it. It had been created under a service account, it was running an unsupported TKGI plan, and it held a 250 GiB persistent volume with the only copy of three years of month end extracts. Rebuilding the wave plan around it cost four days, and the conversation with the data owner about whether that volume had a backup cost considerably more in goodwill than in hours.

Second lesson from the same sweep. We scored admission risk from the Git repositories because that felt rigorous and repeatable, and it reported 12 risky workloads. Running the same question against live pods returned 99. The difference was almost entirely images built from base layers that set USER 0 with no securityContext anywhere in the manifest. Had we carried the number 12 into wave planning, our first wave would have been sized at two weeks and would have run for six.

Verdict: use bosh deployments as your cluster list and treat tkgi clusters only as a name resolution helper. Score admission risk against running pods, never against manifests. And the ordering advice you will read everywhere, that you should migrate dev first because it is safest, is worth ignoring on an estate shaped like this one. Prod carried the highest admission risk ratio at 33 percent, so migrating dev first taught us the least about the problem we actually had. Migrate the noisiest namespace in the least critical cluster first, which for us was search in dev-01, because it exercised hostPath, a DaemonSet and a deprecated autoscaling API in one go.

Inventory First, Design Second

My recommendation for this Part is narrow and firm. Do not draw a single line of your OpenShift target architecture until the nine numbers in the verification section are on a page with your name on it. Every design decision waiting in Parts 6 through 11, how many clusters, which StorageClasses, how Projects map to old plans, how much of NSX-T behaviour has to be rebuilt as NetworkPolicy, is answered by a field in that worksheet. Designing first and discovering second means designing for the estate you imagine.

On Monday, run steps 1 and 2 against your own estate and nothing else. Twenty minutes. Compare the BOSH deployment count against what tkgi clusters shows you. If those two numbers disagree, you have just learned the most important thing this Part has to teach, and you have learned it about your own environment rather than mine.

Part 6 takes this worksheet and turns it into migration waves, using admission risk and storage footprint as the two sorting keys. If you want the wider map first, the TKGI to OpenShift guide lays out all 26 Parts and where this one sits.

References

• Broadcom TechDocs, Retrieving Cluster Credentials and Configuration. Behaviour of tkgi get-credentials, the KUBECONFIG environment variable, and single sign on flags.
• Red Hat, Pod Admission and SCCs Version 2 in OpenShift. Why restricted-v2 drops all capabilities, how the openshift.io/sa.scc.uid-range annotation constrains runAsUser, and when a workload needs nonroot-v2 or anyuid.
• Broadcom KB 446224, Incompatibility between TKGI and VCF 9.1 or NSX 9.1. October 2027 End of Support, and the removal of the legacy NSX Management Plane API that NCP depends on.

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts