Nothing you type on a TKGI cluster turns it into an OpenShift cluster. Written down that looks obvious, and yet the most expensive mistake I keep meeting on these programmes is a plan built on the word upgrade, approved by people who heard upgrade, and funded as though one platform were simply becoming a newer version of itself.

TKGI is a BOSH deployed product, BOSH being the release engineering and VM lifecycle engine that Tanzu Operations Manager fronts with a tile based console. OCP 4 is an Operator deployed product built on Red Hat Enterprise Linux CoreOS. Both speak the Kubernetes API and share almost nothing beneath it. This part covers why that gap has no bridge, what it costs when a programme pretends otherwise, and how to describe the work precisely enough that your change board funds the right shape of it.

Upgrade and migration, defined precisely

Precision here is not pedantry. An upgrade and a migration carry different risk profiles, different rollback mechanisms, different funding models and different failure modes. A committee that approves one will not quietly tolerate the bill for the other.

An in place upgrade preserves cluster identity. Moving a TKGI estate from 1.18 to 1.20 means Ops Manager imports a newer Ubuntu Jammy stemcell, the versioned base image BOSH clones every VM from, applies the changed tile, and rolls masters and workers in turn. Your API endpoint does not change. Existing kubeconfigs keep working. PersistentVolumes stay bound to the same First Class Disks on the same datastore. Applications observe a rolling restart and nothing more. When it goes wrong you revert the tile and pin the previous stemcell.

A platform migration preserves nothing structural. New clusters, new API endpoints, new certificate authorities, new node operating system, new admission controller, new container network, new storage objects, new identity plumbing. Workloads leave as YAML and object storage backups and get recreated somewhere else. Rollback is not a button, it is a DNS record pointed back at a cluster you deliberately kept breathing.

Readers push back at this point, reasonably: surely Kubernetes portability means applications transfer cleanly? Portability holds at the API object level and breaks at everything wrapped around it. A Deployment manifest is a Deployment manifest on both platforms. What differs is whether the resulting pod is admitted, whether its volume binds, whether its ingress resolves, and whether its service account can pull the image. Four independent chances for a lift that looked trivial on a slide to die in a namespace.

Read that as a funding document rather than a technical one. Two rows drive most of the money conversation: concurrency, because you pay for two platforms simultaneously, and duration, because two to four quarters of parallel running lands in a different accounting year to a weekend.

Layers with no conversion path

Take the platforms apart layer by layer and the absence of a conversion path stops being an opinion.

Node operating system. TKGI nodes are Ubuntu Jammy virtual machines produced from a BOSH stemcell. OpenShift control plane nodes run Red Hat Enterprise Linux CoreOS, RHCOS, which Red Hat documents as the only supported operating system for control plane machines. RHCOS is deliberately more constrained than a general RHEL install: /usr is read only, and /etc, /boot and /var are writable but only intended to be altered by the Machine Config Operator, MCO, the component that owns node configuration. Updates arrive as container images that get pulled, extracted, written to disk, and booted into. No procedure exists that turns an Ubuntu stemcell VM into an RHCOS machine. You build new nodes.

Control plane lifecycle. On TKGI a BOSH Director holds desired state for every VM and reconciles toward it. On OpenShift a Cluster Version Operator, CVO, reads a signed release image and drives a set of cluster Operators to the version that image names, while Operator Lifecycle Manager, OLM, handles add on Operators on their own channels. An Operator here means a controller that encodes the operational knowledge for one piece of software. Two incompatible reconciliation engines with two incompatible state stores, and neither can adopt the resources of the other.

Container networking. TKGI on NSX-T uses the NSX Container Plugin, NCP, which watches the Kubernetes API and programs logical switches, routers, load balancers and distributed firewall rules inside NSX. OpenShift 4 ships OVN-Kubernetes as its default container network interface, programming OVN logical flows on each node. Both deliver pod connectivity. They hold intent in different places, express policy through different objects, and cannot be run against the same cluster in sequence.

Pod admission. This layer quietly decides your schedule, and it earns a full part later in the series. TKGI estates typically ran a permissive PodSecurityPolicy posture, PSP being the admission controller Kubernetes removed in 1.25. OpenShift substitutes Security Context Constraints, SCC, and what a service account gets by default is restricted-v2, which drops most Linux capabilities and requires a pod to either leave runAsUser empty or set it inside the range of UIDs allocated to that namespace. An image quietly running as UID 0 on TKGI gets rejected on OpenShift the first day it lands.

Identity. TKGI authenticates through UAA, the Cloud Foundry User Account and Authentication service, usually chained to a corporate LDAP. OpenShift runs its own OAuth server with configurable identity providers. Same directory behind it, different token issuer, different group mapping, and an entirely new set of RoleBindings to write. Five layers, five clean breaks, and not one migrate command among them.

flowchart TB
  subgraph U [Upgrade shape, one platform]
    A[TKGI 1.18 estate] --> B[Ops Manager applies the tile]
    B --> C[Stemcell rolls every node]
    C --> D[Same clusters at 1.20]
  end
  subgraph M [Migration shape, two platforms]
    E[TKGI 1.18 keeps serving traffic] --> F[Build OCP 4 beside it]
    F --> G[Velero backs up one wave]
    G --> H[OADP restores onto OCP]
    H --> I[Fix SCC admission rejections]
    I --> J[Shift DNS and validate]
    J --> K[Repeat for the next wave]
    K --> L[Decommission TKGI and Ops Manager]
  end

Carry over ledger for a TKGI estate

Given all that, the useful question stops being whether things transfer and becomes which ones do. Below is the ledger I hand to application teams at the start of an assessment. Three states only. Carries over means the object moves with little or no change. Rebuilt means the intent survives but the object is recreated in a different form. Discarded means it has no target side equivalent and should never be exported at all. Print it, argue with it against your own estate, and keep the marked up copy. It is the artifact from this part worth returning to.

Two rows deserve a second look, because they are exactly where teams lose weeks.

Ingress objects carry the most dangerous label on that list. OpenShift routers do serve Ingress resources and will generate Route objects from them, so a naive smoke test passes and everyone relaxes. What fails to transfer is the annotation dialect. Every annotation your previous ingress class understood is ignored or handled differently, and the behaviours people genuinely depend on, session affinity, request rewriting, timeouts, TLS passthrough, all live in those annotations. Rewrite ingress by hand rather than trusting the conversion.

Helm charts are the second trap. A chart that pins securityContext.runAsUser to a fixed numeric ID installs cleanly and then fails to schedule under restricted-v2, because that UID sits outside the range allocated to the namespace. Rejection surfaces at pod creation, not at install time, so your pipeline reports a green deploy while the application never comes up. If you take one operational habit from this part, make it that: treat a successful helm install on OpenShift as evidence of nothing.

Budget and schedule effects of upgrade framing

Framing changes numbers. Here is how the reference estate for this series, three TKGI 1.18 clusters on NSX-T carrying 47 namespaces, a Harbor registry and one stateful application with persistent volumes, was scoped under each description of the same work.

Look hard at the first row. Under upgrade framing, parallel capacity is zero, because an upgrade reuses hosts it already owns. Under migration framing you need enough vSphere capacity to stand a complete OpenShift cluster beside running TKGI clusters and hold both until the final wave cuts over. On this estate that meant 18 additional hosts for seven months. Nobody approves that casually, and nobody approves it at all when the paper in front of them says upgrade.

Where those 190 engineer days actually landed is what surprises people most.

Every experienced VMware team I have worked with expects networking to be the hard part, because on TKGI networking was the hard part. NCP misbehaving was the usual cause of an outage, and NSX-T design was where the senior engineers spent their attention. On the OpenShift side networking turned out to be a design exercise with a known answer: take OVN-Kubernetes, expose applications through Routes, translate distributed firewall rules into NetworkPolicy. It fitted inside the 42 day platform build and produced no surprises.

Admission was the surprise. 60 engineer days, close to a third of the programme, spent on pods TKGI had happily run and restricted-v2 refused. Base images running as root, sidecars asking for NET_ADMIN, init containers writing to paths they no longer owned, charts pinning UIDs. None of that is difficult on its own. It is difficult 31 times across 47 namespaces with a different application team, a different release cadence and a different level of enthusiasm behind each one. That is the single biggest correction I would make to how these programmes are usually planned.

Field note from a migration sold as an upgrade

A financial services client with three TKGI clusters took their paper to a technology investment board in the spring. That paper said platform upgrade. Partly because the author sincerely believed a Kubernetes estate moving to another Kubernetes distribution counted as an upgrade, and partly because upgrade fitted an approved budget line while migration required a fresh capital case. Board approved it in twenty minutes.

Eleven weeks later we needed hosts. Standing up an OpenShift cluster with three control plane nodes and six workers, sized to eventually hold everything the TKGI estate held, is not something you carve out of spare capacity in a production vSphere cluster already running the platform you are trying to leave. Our capacity request went back to the board and came back rejected, not because it was unreasonable but because it did not match the approved paper. An upgrade does not need new hosts. We had told them it was an upgrade.

Six weeks went to re-approval. Two of those weeks were spent writing an honest paper. Four were spent waiting for the next board cycle. Throughout, the migration team was fully staffed and largely idle, which cost considerably more than the hosts would have, while the TKGI clusters kept running and kept accruing exactly the support risk we had been funded to retire.

Something else broke in the same period, and it taught me more. Because the framing said upgrade, application teams had been told to expect a rolling restart. So when we asked them for maintenance windows for the first wave, we got pushback, and fairly: you told us this was transparent. We had set an expectation the architecture could never meet. Rebuilding that trust took longer than rebuilding the schedule, and on the stateful application we eventually negotiated a 45 minute window that would have been granted for free in month one had we simply asked honestly.

Reversed decision, stated plainly. I now refuse to write a migration paper containing the word upgrade anywhere, including in sections where it would be technically defensible, such as the OpenShift minor version updates you will run afterwards. Different words for different work, from the first slide onward. Small discipline, and it has protected every programme since.

Plan it as a rebuild with data movement

Best sentence I have found for describing this work to a non technical board: you are building a second Kubernetes platform, moving applications onto it in waves, and switching the old one off. No jargon, no false comfort, and every cost line follows naturally from it.

Practically, three things belong in your plan that an upgrade plan would never contain. A parallel running period with a funded end date, so nobody discovers in month nine that they are paying for two platforms indefinitely. A per wave rollback that is a traffic decision rather than a restore, which only works if the source cluster stays alive and untouched until validation passes. And an admission remediation workstream with its own owner and its own budget, sized from a real dry run instead of a guess.

Should you be reading this series at all, or should you be looking at VMware vSphere Kubernetes Service on VCF 9? Legitimate fork in the road, and it gets a fair treatment in the TKGI to VKS complete guide. Briefly: if your organisation intends to stay inside the VMware operating model and has a VCF 9 path, VKS is the shorter journey. If you are leaving the VMware platform stack, or you already run OpenShift elsewhere, or your applications are heading toward Operators and OpenShift AI workloads, this series is yours. The full map of what is coming sits on the TKGI to OpenShift migration guide.

Your Monday action: open the change record from your last TKGI upgrade, 1.17 to 1.18, and count the tasks in it. Then walk the carry over ledger above against one production namespace and count what has to change. When those two numbers land within an order of magnitude of each other, your plan is still an upgrade plan, and it will not survive contact with restricted-v2. Part 4 puts money on this: subscription costs, the parallel running bill, and a phased timeline you can take to a board without the word upgrade appearing once.

References

• Red Hat, About the Migration Toolkit for Containers, on installing the MTC Operator on all clusters and migrating SCC linked resources.
• Red Hat, Red Hat Enterprise Linux CoreOS (RHCOS), OpenShift Container Platform 4.18 Architecture, on read only /usr and Machine Config Operator ownership.
• Broadcom, Upgrading Tanzu Kubernetes Grid Integrated Edition with Ops Manager, TKGI 1.20, on the stemcell and tile based in place upgrade path.
• Red Hat, OADP application backup and restore, OpenShift Container Platform 4.19, on Velero packaging and cross cluster restore.

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts