Decommissioning is the only phase of this migration that cannot be undone, which makes it the only phase that should be deliberately slow. Everything up to this point was reversible by pointing a DNS record back at the old load balancer. Once tkgi delete-cluster returns, that cluster, its etcd, its node VMs and most of its NSX-T objects are gone, and your only route back is a restore into a platform you are actively dismantling. Twenty five parts of this series were about building something. This one is about proving you no longer need the thing you are about to destroy.

Teardown Order That Protects Your Rollback

A TKGI estate is a stack of dependencies, and each layer can only be removed by the layer above it. Clusters are BOSH deployments owned by the TKGI tile. Once you delete that tile, no supported path exists to remove a cluster that is still running, and you are left deleting VMs by hand in vCenter while NSX-T keeps every logical router the NSX Container Plugin (NCP) ever created. Ops Manager, in turn, owns the BOSH Director, so removing the Ops Manager VM before you have deleted the installation strands the Director with nothing to manage it. Work top down, and every step still has a supported command behind it. Work in any other order and the rest of the teardown becomes manual archaeology.

That chart is worth reading before you plan the sequence, because it tells you where the impatience will come from. Production alone accounts for 78 of the 142 vCPU, and finance will notice when it comes back. Everything below it is rounding error. There is no capacity argument for rushing the small stages, and there is a very good reason to leave the development cluster running long past the point where it feels justified. Four VMs and 16 vCPU is the cheapest insurance policy in this entire migration. It is also the only remaining place where a forgotten pipeline can fail loudly instead of silently.

Preflight, Proving Nothing Still Depends on TKGI

Asking application teams whether they still use TKGI produces a confident yes from people who have not looked, and a confident no from people who have forgotten. Ask the platform instead. Three sources give an honest answer: Kubernetes audit logs for who is still calling the API server, NSX-T load balancer statistics for who is still hitting the old virtual servers, and your DNS resolver logs for who is still resolving the legacy hostnames. My bar for declaring a cluster quiet is thirty consecutive days with no non-monitoring API request and zero bytes through the ingress virtual servers. Anything short of that is a guess dressed up as a decision.

Seven ingress objects on a cluster that supposedly serves nothing is exactly the result you want from preflight, because it is cheap to investigate now and expensive to investigate after deletion. In our estate six of those seven were stale definitions left behind by teams who had already moved to Routes, and the seventh was a monitoring endpoint that a partner had bookmarked. That last line of output is the more interesting one. A Concourse worker service account was still authenticating against production, which turned out to be a contract test suite nobody had reassigned. Fix that class of thing while the cluster exists. It is a ten minute change now and an incident later.

Steps 1 to 5, Draining and Deleting TKGI Clusters

flowchart TD
  A[Ninety day quiet period after cutover] --> B[Final Velero backup, long TTL]
  B --> C[Delete staging cluster]
  C --> D[Delete production cluster]
  D --> E[Delete development cluster last, the tripwire]
  E --> F[Export Ops Manager installation settings]
  F --> G[Delete TKGI tile and apply changes]
  G --> H[Delete BOSH Director and Ops Manager VM]
  H --> I[Sweep NSX T1 routers, IP blocks and VIPs]
  I --> J[Sweep orphan vSphere First Class Disks]
  J --> K[Release subscription, archive backup bucket]

Steps 1 to 3 are preparation inside each cluster, and skipping any of them turns a clean deletion into a support case. PodDisruptionBudgets block the drain that BOSH performs as it removes worker VMs, so a single budget with minAvailable set to the full replica count will hang the delete indefinitely. PersistentVolumes carrying a Retain reclaim policy are worse, because deletion succeeds and quietly leaves the backing disk on the datastore with no Kubernetes object referencing it. Capture that list of volume names before you delete anything, because after the cluster is gone the only identifier left on the disk is a UUID.

Step 5 is to leave dev-01 alone. Resist the tidiness instinct here. On our estate the development cluster stayed up for the full ninety days after production went, and it earned its keep in the first fortnight. That is also the point at which the happy path ends, because production did not delete cleanly. It sat in progress for two hours and eleven minutes, and the TKGI CLI told me nothing useful. BOSH did.

That error is the single most common way a TKGI teardown stalls, and it is worth understanding rather than working around. NCP creates a tier 1 logical router per cluster and hangs the load balancer virtual servers for every LoadBalancer service off it. When ingress objects are deleted out of band, by a person tidying up or by a Velero restore that removed the service on the source side, NCP loses track of the virtual server but NSX-T keeps it attached. BOSH then cannot delete a router that still has ports on it, and it retries for an hour before giving up. Fix it in NSX-T Manager under Networking and then Load Balancing: find the virtual server and server pool bound to that tier 1 router, delete both, then rerun tkgi delete-cluster prod-01 –wait. Ours completed in nine minutes on the second attempt. Broadcom documents the same cleanup stall against Supervisor clusters, and the remedy is identical.

Steps 6 to 10, Ops Manager, BOSH and NSX-T Teardown

Before anything destructive at this layer, export the Ops Manager installation settings. That zip file is a complete record of network ranges, availability zone mappings, plan definitions, certificate subjects and tile configuration, and it is the only artifact that can answer an auditor asking what the old estate looked like eighteen months from now. It costs one command and a few megabytes. I have never regretted taking it, and I have twice regretted not taking it earlier. Store it outside the vSphere estate you are about to shrink, and never in the same bucket as your Velero backups.

Step 9 removes Ops Manager itself, and the order inside this one step matters more than any other in the runbook. Delete the installation from the Ops Manager dashboard first, using Delete This Installation, and let it remove the BOSH Director and its VMs while something still owns them. Only then power off and delete the Ops Manager VM in vCenter. If you deployed the Director directly with bosh create-env rather than through Ops Manager, the equivalent is bosh delete-env against the same manifest and state file, which idempotently removes the VMs, disks and stemcells it created. Reverse this order and you get a Director VM that nothing manages, running BOSH agents that nothing talks to, which is a slow afternoon in vCenter with a naming convention as your only guide.

Forty one detached disks totalling 2.1 TB is not an unusual result, and it is the reason step 3 exists. Every one of those disks was created by the vSphere Container Storage Interface (CSI) driver for a PersistentVolume with a Retain reclaim policy, which is precisely the policy a careful operator sets on database storage. Careful behaviour during the build becomes cleanup debt during the teardown. Match UUIDs against the file you saved, keep the ones backing anything you might still restore, and delete the rest deliberately rather than in a loop. NSX-T deserves the same treatment: check for tier 1 routers, IP blocks, SNAT pools and virtual servers whose names still carry a cluster UUID, and remove them only after confirming no other consumer shares the block.

Verification and Clean End State

Green here looks like a series of commands failing in the right way. A BOSH client that cannot find a Director, a vCenter search that returns nothing, an NSX-T query with a zero count. Run all of them, and run them from the same jump host your automation used, because a leftover credential in a CI secret store is its own kind of debt.

Rollback at this stage is no longer a platform operation, it is a data operation, and it only works if you kept the right things. Below is the artifact I would hand a colleague on their first decommission. It is short on purpose, and the only two rows people argue with are the backup bucket and the exported settings, both of which are cheap to keep and impossible to recreate.

Common Failures and Remediation

Ninety Day Rollback Window, Then Delete

My recommendation for this final phase is a ninety day quiet period, measured from the last production traffic shift rather than from the change ticket, with the development cluster left running for its whole length. Ninety days covers a monthly close, a quarterly report and at least one seasonal load pattern, which between them surface the dependencies that a fortnight never will. Against that, the carrying cost is four VMs and 16 vCPU. I would not run a shorter window on a regulated estate, and I would not run a longer one anywhere, because a cluster nobody patches becomes a liability faster than it becomes a safety net. TKGI reaches official End of Support in October 2027, so this window is not open indefinitely in any case.

What comes next depends on what pushed you here. If the driver was NSX 9.1 and VCF 9, the OpenShift landing is one of two valid answers and the other one is worth reading before you commit anything else to it, which is why the TKGI to VKS series exists alongside this one. If the driver was consolidation, the next milestone is fleet management with Red Hat Advanced Cluster Management rather than another cluster. And if the pressure now coming at your platform is GPU workloads and model serving, OpenShift AI is the natural next build, covered end to end in the Red Hat Gen AI guide. Whichever way you go, the platform you just built is the substrate for all three.

Your Monday action is small and it starts the clock. Run the ingress count and the API audit query from the preflight section against all three TKGI clusters, write down the date of the last non-monitoring request on each, and put ninety days from that date in the calendar as the teardown start. That single date turns decommissioning from an argument into a schedule. When it arrives, work down the ten steps above in order, and keep the bucket. Twenty six parts on, the whole series is collected in the TKGI to OpenShift complete guide, so start wherever your estate actually is.

References

• Broadcom, Deleting Clusters, Tanzu Kubernetes Grid Integrated Edition 1.18
• Broadcom, Delete Your Tanzu Kubernetes Grid Integrated Edition Deployment
• Broadcom KB 446224, Incompatibility between TKGI and VCF 9.1 or NSX 9.1
• Velero, How Velero Works, backup expiry and TTL
• Cloud Foundry BOSH, CLI v2 command reference

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts