Every target architecture I have drawn for a TKGI estate began life as a copy of the estate it was replacing, and every one of them got smaller on the second pass. Our reference estate runs three Tanzu Kubernetes Grid Integrated (TKGI) clusters across two vSphere datacenters, and the honest OpenShift target is two clusters, because a TKGI cluster boundary usually encodes a Plan and a BOSH deployment rather than a real isolation requirement. This Part turns ten Parts of assessment into vCenter objects, machine counts and StorageClass names you can hand to whoever builds it.

Design Inputs Carried Over from TKGI

Six Parts of assessment produce a short list of inputs, and most of this architecture is arithmetic on that list. Part 5 gave us the estate inventory. Part 6 sorted applications into migration waves. Part 7 established that admission control, not networking, is where a lift actually breaks. Part 8 mapped NCP, the NSX Container Plugin that wires TKGI pods into NSX-T, onto OVN-Kubernetes. Part 9 mapped persistent volumes onto vSphere CSI, and Part 10 mapped UAA and LDAP onto OpenShift OAuth and Projects.

Three of those inputs dominate everything downstream, and they need answering in order. How many clusters do we build. Where do their machines sit in vSphere. What is a worker node actually sized for. Get the first one wrong and the other two are wrong at a multiple.

Part 2 established the inversion that governs all three answers: TKGI is a cluster factory, and OpenShift 4 is a self managing cluster. On TKGI a new cluster cost you a handful of virtual machines, because Ops Manager, the BOSH director, the NSX-T integration and the upgrade pipeline were all shared infrastructure sitting outside every cluster. On OpenShift none of that is shared. Each cluster brings its own control plane, its own Cluster Version Operator, its own monitoring and logging stacks, its own image registry and its own upgrade calendar that somebody has to schedule and watch. A cluster stopped being cheap the moment you left TKGI, and every design decision below falls out of that.

One input deliberately does not carry over. TKGI Plan definitions look like sizing decisions and they are not, they are packaging decisions from an era when a Plan was the only way to give a team a different node shape. On OpenShift a node shape is a MachineSet, and a MachineSet costs nothing to add, so do not port your Plan catalogue. Rebuild node shapes from what workloads actually request, which Part 6 already measured.

Cluster Topology, Consolidating Three TKGI Clusters into Two

Our reference estate today is three TKGI clusters. Dev runs a single master and four workers. Staging runs three masters and five workers. Production runs three masters and twelve workers. Add the Ops Manager appliance and the BOSH director and the estate is 30 virtual machines, of which 21 do any application work.

Dev and staging were separate clusters for reasons that stopped applying. A TKGI Plan change is cluster wide, so anyone who wanted a different node shape got a different cluster, and BOSH made that almost free. Neither reason survives the move. Apply the same evidence test we used for tenancy in Part 10: is there a regulator, an auditor or a contract that requires separation, or is this a preference that grew a habit. For dev and staging in our estate the answer was preference, so they merge into one nonprod cluster with a Project per environment.

Merging all the way down to one cluster is where I stop, and it is worth saying why, because the consolidation logic seems to point there. OpenShift minor upgrades are a real event with a real failure mode, and you want a cluster that takes an upgrade a week or two before production sees the same version. Two clusters buys you that rehearsal. One cluster means your rehearsal is production. Going the other way, a fourth cluster costs three control plane machines, two or three infrastructure machines and a separate upgrade slot in somebody calendar every four months, which is the cadence Red Hat ships minor releases at.

Read the last line of that chart carefully, because it changes the business case Part 4 built. Machines carrying the infra role are excluded from the subscription count even when the worker role is also applied, and control plane machines have never counted. So a 27 machine estate is a 16 machine subscription, provided you actually create the infrastructure machine sets and label them rather than letting ingress and monitoring land on general workers.

vSphere Placement, Failure Domains and Datastore Layout

A failure domain in OpenShift is a named bundle of vCenter objects, a datacenter plus a compute cluster plus a datastore plus one or more port groups, optionally a resource pool and a folder. Declare several of them and the installer spreads control plane machines across them, the vSphere cloud provider tags nodes with topology.kubernetes.io/zone and topology.kubernetes.io/region, and CSI provisioning becomes topology aware. Declare none and every machine lands in one compute cluster on one datastore, which is exactly the posture that made a TKGI master outage a full cluster outage.

Note what that table separates. Control plane machines get their own small fast datastores, and application persistent volumes go somewhere else entirely. That split exists for one reason, covered in the field note below, and it is not negotiable in a production design.

Here is what a copy paste mistake in that block actually costs you, and it is cheap only because it fails before anything is built:

Node Roles and Sizing, Control Plane, Infra and Workers

OpenShift gives you three node roles where TKGI gave you two. Control plane machines run the API server and etcd. Worker machines run applications. Infrastructure machines are workers wearing an infra label, carrying ingress routers, the monitoring stack, the internal image registry and logging, and they are excluded from your subscription count. Red Hat recommends at least three infrastructure machine sets in a production deployment, partly for spread and partly because several add on components want three replicas on distinct nodes.

Two of those numbers deserve defending. Nonprod runs two infrastructure nodes rather than three, which knowingly departs from Red Hat production guidance, and I am comfortable with that because nonprod does not run the logging stack and can survive an ingress router losing a replica for an hour. Production keeps three. Separately, that 4 vCPU and 16 GB nonprod control plane is the documented minimum for a control plane machine, and it is a minimum, not a recommendation. Production gets 8 and 32 because etcd behaves much better with headroom.

Now the sizing argument that matters most, because it is where TKGI habits cost real money. Do not port your TKGI worker shape one for one. Every OpenShift node runs a fixed set of platform daemonsets, OVN-Kubernetes, the Machine Config daemon, monitoring agents, the CSI node plugin, and that overhead is per node, not per pod:

Run that arithmetic against a plan to lift 21 TKGI workers into 21 OpenShift workers of 8 vCPU and 32 GB. You would burn about 21 vCPU and 65 GB on daemonsets, roughly 12 percent of the estate, to buy nothing. Sixteen larger workers spend about 16 vCPU and 50 GB on the same overhead while offering bigger schedulable holes, which matters because our PostgreSQL tier requests 8 GB and will not fit a fragmented small node. Fewer and larger wins until you hit blast radius, which for us is around 20 nodes per cluster.

On storage totals, Red Hat documents a floor of 800 GB for a standard vSphere cluster. Our production layout lands at roughly 2.7 TB of machine disk before a single persistent volume exists. Treat the documented figure as a validation floor, never as a capacity plan.

Platform Decisions in One Mapping Table

Here is the artifact worth keeping from this Part. Every component of our TKGI estate, what replaces it on OpenShift, the decision we made, and which later Part actually builds it. Print it, put it in the design document, and use it as the migration checklist.

If your organisation is weighing VMware vSphere Kubernetes Service on VCF 9 instead of OpenShift, that is a legitimate fork in the road and a different design entirely, covered in the TKGI to VKS Complete Guide. Everything from here forward assumes OpenShift.

Field Note from Sizing a Replacement Estate

I put our first production control plane on the biggest, most convenient datastore we had, a shared vSAN volume that was already carrying database virtual machines. It installed cleanly. Everything looked fine for about nine days.

Then etcd started logging leader changes, a few per hour at first, then during a nightly batch window enough of them that oc commands intermittently timed out. Red Hat guidance here is specific and I had ignored it: the 99th percentile of etcd_disk_wal_fsync_duration_seconds should stay under 10 ms, and etcd wants a block device that can sustain 50 IOPS of 8 KB sequential writes including fdatasync in under 10 ms. Our shared datastore was nowhere near that once the batch job started.

Average fsync went from 27 ms to 3 ms and leader changes stopped that night. Cost of the mistake: two days of investigation and a maintenance window I had already told the change board we would not need. Cost of avoiding it: running one fio command per candidate datastore during design, which takes about a minute each.

Second reversal from the same design, smaller but more embarrassing. I had originally drawn four OpenShift clusters, one per TKGI cluster plus a management cluster for Red Hat Advanced Cluster Management. We built the management cluster, watched it sit at roughly 4 percent utilisation across six machines for two months managing exactly two spokes, and folded RHACM onto the nonprod cluster instead. RHACM runs hub and spoke and its hub is itself an OpenShift cluster, so a dedicated hub is real infrastructure you keep forever. Below about five managed clusters it is not worth the machines.

Build Two Clusters and Declare Zones on Day Zero

My recommendation for this estate, and for most three cluster TKGI estates that look like it: two OpenShift clusters, IPI installed, three vSphere failure domains declared in install-config.yaml from the first attempt, control plane machines on their own fast datastores, three infrastructure machines in production carrying ingress and monitoring, and workers at 16 vCPU and 64 GB rather than a copy of your TKGI worker shape. Skip the dedicated RHACM hub until you are managing five or more clusters.

Avoid two things specifically. Avoid a one for one cluster mapping from TKGI, because you will pay for three control planes and three upgrade calendars to preserve a boundary that BOSH created for its own convenience. And avoid the flat single zone install with a plan to add zones later, because that plan quietly means rebuilding the cluster.

Your Monday task: open vCenter, and for each candidate datastore that might hold control plane machines, run the fio command above against a scratch virtual machine on it. Write the 99th percentile figure next to the datastore name in your design document. If none of them comes in under 10 ms, you have found the first real blocker in your migration, and you have found it in design rather than in week two of production. Part 12 takes this design and installs it, comparing IPI against UPI on vSphere and explaining which one this estate should pick.

References

Red Hat, Multiple regions and zones configuration for a cluster on VMware vSphere, OpenShift Container Platform 4.18

Red Hat, Recommended etcd practices, OpenShift Container Platform

Red Hat, Creating infrastructure machine sets, Machine management, OpenShift Container Platform 4.18

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts