, ,

TKGI and OpenShift Architecture Compared, Component by Component (TKGI to OpenShift Series, Part 2)

A component by component architecture comparison of TKGI and OpenShift Container Platform 4, covering control planes, node lifecycle, networking, admission control and the operational trade offs that decide migration effort.

TKGI to OpenShift Series · Part 2 of 26
$ oc get machines -n openshift-machine-api NAME PHASE TYPE REGION ZONE AGE ocp1-h5d2t-master-0 Running 36d ocp1-h5d2t-master-1 Running 36d ocp1-h5d2t-master-2 Running 36d ocp1-h5d2t-worker-9xq4z Running 36d ocp1-h5d2t-worker-c7ktp Running 36d

That output is where the two platforms stop being comparable. A Kubernetes API is listing the virtual machines it runs on, in vCenter, as first class objects it owns. On Tanzu Kubernetes Grid Integrated (TKGI, formerly Enterprise PKS) nothing inside a cluster knows its own virtual machines exist, because BOSH owns them from outside and the cluster is only a consequence. Almost every difference in this comparison falls out of that one inversion, and so does almost every hour of migration effort.

Who this is for: A TKGI operator who knows Ops Manager, BOSH and NSX Container Plugin (NCP) well and needs an accurate mental model of OpenShift Container Platform 4 (OCP) before designing a target. Terms defined on first use here: BOSH is the release and virtual machine lifecycle engine TKGI inherits from Cloud Foundry; Ops Manager is the tile based console that installs BOSH managed products; RHCOS is Red Hat Enterprise Linux CoreOS, the immutable operating system OpenShift nodes run; MCO is the Machine Config Operator; OLM is Operator Lifecycle Manager; SCC is a Security Context Constraint; OVN-Kubernetes is the default OpenShift container network plugin; CSI is the Container Storage Interface.
Key takeaways: TKGI puts cluster lifecycle outside the cluster, in Ops Manager and BOSH. OpenShift puts it inside, in the Cluster Version Operator and roughly thirty cluster Operators that reconcile continuously. Node operating system management inverts the same way: BOSH stemcells are pushed onto virtual machines, RHCOS is rendered by the MCO from objects that live in the API. Networking and storage map with design work, ingress maps with a DNS plan, but admission control does not map at all, and that is the line item that decides your migration schedule. Read the mapping table below as the artifact, not the prose.

Two control planes, side by side

A TKGI installation is a stack of things that manage the thing below them. Ops Manager installs and configures BOSH Director. BOSH Director deploys the TKGI control plane, which Broadcom documents as a pair of virtual machine groups: a TKGI API virtual machine group hosting the TKGI API, the UAA identity server and the TKGI Broker, and a TKGI Database cluster holding cluster management state. When a developer asks for a cluster, that control plane hands the request to BOSH, BOSH provisions master and worker virtual machines from a stemcell, and NCP wires the resulting pods into NSX-T. NCP, the NSX OSB Proxy and the BOSH CPI are all internal TKGI components. You never see them as Kubernetes objects because they are not Kubernetes objects.

OpenShift collapses that stack into the cluster. After installation there is no external daemon holding desired state. A Cluster Version Operator watches the release payload and reconciles a set of cluster Operators that own the API server, etcd, ingress, networking, storage, monitoring, authentication and the machine API. Alongside it, OLM manages optional add on Operators and is itself composed of two Operators, an OLM Operator that deploys applications defined by ClusterServiceVersion resources and a Catalog Operator that tracks catalogues and channels. Red Hat documents this split plainly: cluster Operators are managed by the Cluster Version Operator and installed by default, while add on Operators are managed by OLM. If you want to know what your platform is doing, you query it.

$ oc get clusteroperators NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.18.9 True False False 21d etcd 4.18.9 True False False 36d ingress 4.18.9 True False False 36d machine-config 4.18.9 True False False 3h14m network 4.18.9 True False False 36d storage 4.18.9 True False False 36d

Versions tested against for this comparison: TKGI 1.18 on NSX-T with NCP, OpenShift Container Platform 4.18.9 installed IPI on vSphere 8.0 U3, oc client 4.18.9. Nothing above is a procedure, it is a shape. Hold on to the shape and the rest of the series will make sense.

One consequence of that shape catches every TKGI team, and it is worth stating before the design workshop rather than after it. On TKGI, tkgi clusters gives you a fleet view because a fleet manager exists by definition. OpenShift ships no equivalent. A cluster knows about itself and nothing else, and multi cluster views come from Red Hat Advanced Cluster Management, a separate subscription with its own hub cluster. Teams running three TKGI clusters tend to assume they will run three OpenShift clusters, discover this halfway through the design, and then reopen the budget. Decide fleet strategy in the same session where you decide tenancy, because the honest options are few large clusters carved into Projects with no hub, or many small clusters with a hub you pay for. Choosing many clusters out of habit is the expensive default.

flowchart TD
  subgraph TKGI [TKGI ownership chain]
    A[Ops Manager tile] --> B[BOSH Director]
    B --> C[TKGI API and Database VMs]
    C --> D[Cluster master and worker VMs]
    B --> D
    D --> E[NCP wires pods into NSX-T]
  end
  subgraph OCP [OpenShift ownership chain]
    F[Release image] --> G[Cluster Version Operator]
    G --> H[Cluster Operators]
    H --> I[Machine Config Operator]
    I --> J[RHCOS nodes]
    H --> K[Machine API talks to vCenter]
    G --> L[OLM manages add on Operators]
  end
Lifecycle authority sits outside the cluster on TKGI and inside the cluster on OpenShift. Every mapping decision later in the series inherits from this.

Node lifecycle, stemcells against RHCOS

On TKGI you patch nodes by importing a new stemcell into Ops Manager and clicking Apply Changes. BOSH recreates virtual machines in a rolling order it decides, using canaries and max in flight values from the deployment manifest. The unit of change is a virtual machine image plus a release, and the operator experience is a progress log in a web console.

OpenShift nodes run RHCOS, which Red Hat documents as the only supported operating system for control plane machines. Its configuration is delivered by Ignition at first boot and afterwards by MachineConfig objects. Write a MachineConfig, and the MCO renders it into a single rendered config per machine config pool, then a daemon on each node applies it, cordons, drains and reboots. The unit of change is a Kubernetes object, and the operator experience is watching a pool report updated machine counts.

Where the obvious approach is wrong: every TKGI operator I have moved across keeps the reflex of logging in to fix a node, because on TKGI a quick bosh ssh and a config edit was normal and survived until the next deploy. On RHCOS it does not survive, and worse, a hand edited file makes the machine config daemon mark the node degraded and stall the whole pool on the next roll. If a change is worth making it is worth being a MachineConfig. The second reflex worth dropping is accepting the default maxUnavailable of 1 on the worker pool. On a 40 worker migration cluster that turns a routine config roll into most of a working day.

Component by component mapping

This table is the artifact worth keeping from this part. Print it, take it into the design workshop, and use the last column to work out who on your team needs retraining. The point of the fourth column is that it names what an operator actually types, which is a better predictor of retraining cost than any architecture diagram.

ConcernTKGI 1.18OpenShift 4What you type on OpenShift
Platform installOps Manager tile plus BOSH Directoropenshift-install with an install configopenshift-install create cluster
Platform state reconciliationBOSH deployment manifests, applied on demandCluster Version Operator plus cluster Operators, continuousoc get clusteroperators
Node operating systemBOSH stemcell, mutable at runtimeRHCOS, immutable, rendered by the MCOoc get mcp and oc apply MachineConfig
Node pools and sizingTKGI plans, defined in the tileMachineSets, defined as cluster objectsoc scale machineset
Cluster provisioning for teamstkgi create-cluster, one cluster per tenantProjects inside one cluster, quotas and limitsoc new-project and oc apply ResourceQuota
IdentityUAA on the TKGI API VM, backed by LDAPOpenShift OAuth server with identity providersoc edit oauth cluster
Container networkNSX-T with NCP, one T1 per namespaceOVN-Kubernetes overlay using Geneveoc get network.operator cluster
Ingress and load balancingNSX-T load balancer created by NCPIngress Controller and Route objectsoc get routes and oc get ingresscontroller
MicrosegmentationNSX-T distributed firewall, often managed outside KubernetesNetworkPolicy and AdminNetworkPolicy in the clusteroc get networkpolicy -A
Persistent storagevSphere CSI driver installed by the tilevSphere CSI driver owned by a cluster Operatoroc get sc and oc get csidrivers
Image registryHarbor, deployed as its own tileInternal registry, Harbor optional as a mirroroc get config.imageregistry cluster
Pod admissionPodSecurityPolicy, usually permissive in practiceSCC, restricted-v2 by default for all usersoc get scc and oc adm policy
Backup and restoreBBR for the management plane, Velero for workloadsOADP, which packages Velerooc get backups.velero.io -n openshift-adp
MonitoringBring your own, commonly a bolted on PrometheusCluster monitoring stack shipped and supportedoc get pods -n openshift-monitoring

Two rows in that table are doing more work than the others. Cluster provisioning changes shape completely: a TKGI shop that hands every team its own cluster usually consolidates onto Projects, and that is a tenancy redesign rather than a migration step. Microsegmentation changes owner: firewall rules that lived with the network team in the NSX-T distributed firewall become NetworkPolicy objects in a namespace, owned by whoever owns the namespace. Both are covered properly later in this series, and neither should be discovered during a cutover weekend.

Two more rows deserve a sentence each because they look harmless and are not. Registry is the friendlier of the pair: OpenShift ships an internal registry owned by a cluster Operator, and Harbor keeps working as an upstream mirror through the transition, so image pull paths change in manifests but nothing has to be rebuilt on day one. Storage looks identical on both sides because both run the vSphere CSI driver, and that similarity hides the real difference. On TKGI the driver arrived with the tile and StorageClass names were yours to pick. On OpenShift a cluster Operator owns the driver and installs its own default StorageClass, so a PersistentVolumeClaim restored from a Velero backup naming a class that does not exist here will sit in Pending until somebody notices at two in the morning. Standardising storage class names is the cheapest preparation you can do before a first restore, and Part 9 spends its whole length on that problem.

Admission control, where the architectures actually diverge

Everything above maps with design effort. Admission does not map, it replaces. Red Hat grants the restricted-v2 SCC to all authenticated users by default, and restricted-v2 constrains the user a container runs as with a MustRunAsRange strategy. Crucially, the SCC itself does not carry the range. The admission plugin reads the openshift.io/sa.scc.uid-range annotation on the project and populates the range from there, so every project gets a different, effectively unpredictable starting user id. A workload that ran fine on TKGI because your PodSecurityPolicy posture never really constrained it now meets an admission controller that does.

$ oc get events -n webapp –field-selector reason=FailedCreate Error creating: pods "web-6f8c9d4b7-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, spec.containers[0].securityContext.runAsUser: Invalid value: 0: must be in the ranges: [1000700000, 1000709999]] $ oc get project webapp -o jsonpath='{.metadata.annotations.openshift.io/sa.scc.uid-range}’ 1000700000/10000

Three fixes exist and only one of them is good. Best: rebuild the image so it does not need user 0 and leave runAsUser unset, which lets the range default apply. Acceptable under time pressure: grant the nonroot-v2 SCC to that specific service account, not to the namespace and never to default. Bad, and the one people reach for first: grant anyuid, which quietly returns your OpenShift cluster to the permissive posture you left behind on TKGI. If your security team signed off on the migration partly on the strength of OpenShift admission, granting anyuid across the estate is how you lose that argument in the first audit.

Operational trade offs, measured

AxisStronger on TKGIStronger on OpenShiftVerdict
Cluster provisioning speedOne CLI call against a plan, minutes to a running clusterFull installer run, longer per clusterTKGI wins, but the answer on OpenShift is fewer clusters
Platform self healingBOSH resurrector restores VMs, not configuration driftOperators reconcile configuration continuouslyOpenShift wins clearly
Network integration depthNative NSX-T objects per namespace, one policy planeOverlay independent of the NSX version you runTKGI wins on depth, OpenShift wins on future optionality
Default security postureNothing, PSP posture is whatever you configuredrestricted-v2 applies to every authenticated userOpenShift wins, and it is also your biggest migration cost
Ecosystem install and upgradeExtra tiles, each with its own upgrade windowOLM subscriptions with channels and approval gatesOpenShift wins clearly
Skills already in the buildingYour VMware and BOSH knowledge transfers directlyKubernetes knowledge transfers, BOSH knowledge does notTKGI wins today, and this is why VKS deserves a look

Rolling change is the axis nobody costs properly, so I measured it on the reference estate: three TKGI 1.18 clusters and one OpenShift 4.18 cluster, twelve workers each, same vSphere hosts, same storage. Numbers below are wall clock from command to fully rolled, averaged over three runs.

Rolling a change across twelve worker nodes Wall clock minutes, averaged over three runs on the reference estate. TKGI 1.18 and OCP 4.18.9 on the same vSphere hosts. TKGI apply changes 214 min TKGI stemcell only 96 min OCP maxUnavailable 1 128 min OCP maxUnavailable 3 52 min 0 50 100 150 200 Lower is better. OpenShift only wins this axis once you change the default.

Read that chart carefully, because the headline is not that OpenShift is faster. At the shipped default it is slower than a plain TKGI stemcell roll, 128 minutes against 96. It becomes faster only after you raise maxUnavailable on the worker pool, at which point it beats a full Ops Manager Apply Changes by a factor of four. Platform speed on OpenShift is a configuration decision, not a property, and it is one of the few tuning knobs I would set before the first workload lands rather than after.

A note on tooling belongs here too, because it saves an argument later. Both kubectl and oc work against an OpenShift API server, and I use oc throughout this series for one reason: oc knows about object types that have no upstream equivalent. Routes, Projects, SCCs, ImageStreams and BuildConfigs are all first class here, and oc carries admin verbs such as oc adm policy and oc adm must-gather that have no kubectl form at all. Running kubectl against OpenShift is not wrong, it simply means half of your runbook falls back to raw resource names and hand written yaml, and the half that breaks under pressure in production is usually the admin half.

Field note from a side by side upgrade test

I lost most of a Thursday to this comparison, and it was my own fault. We were running the two platforms in parallel for a bank client and I wanted a fair upgrade timing, so I pushed a MachineConfig that added a chrony entry to the worker pool, then went to a meeting. When I came back the pool showed 4 of 12 machines updated and no progress for 90 minutes. Nothing in the events said why. It turned out that two nodes had a PodDisruptionBudget on a legacy application that permitted zero disruptions, a copy of the exact PDB the team had carried across from TKGI where BOSH had happily ignored it. MCO does not ignore it. It waits, politely, forever.

Total cost was about four hours, most of it spent looking in the wrong place because I assumed the machine config itself was bad. That incident reversed a decision for us: we had planned to move PodDisruptionBudget objects across untouched as part of the workload manifests, on the reasonable grounds that they are standard Kubernetes. We now review every PDB during assessment and reject any with minAvailable equal to the replica count. It costs an hour per application wave and it has saved that Thursday from happening again on three engagements.

Worth knowing: objects that were inert on TKGI can become load bearing on OpenShift. PodDisruptionBudget is one, because MCO honours it during node drains. Pod priority classes are another, because OpenShift ships its own and a workload priority set too high will preempt platform components. Neither shows up in an architecture diagram, and both belong in your assessment checklist in Part 5.

Architecture verdict for a TKGI shop

Judged purely as architecture, OpenShift is the stronger platform for a team that wants Kubernetes to be self contained: one reconciliation model, one place to look, an ecosystem installed through OLM rather than through more tiles, and a default admission posture your auditors will like. TKGI keeps two genuine advantages, per cluster provisioning speed and the fact that your existing NSX-T and BOSH skills already work. If those two advantages dominate your thinking, that is a signal worth taking seriously, and it usually means the honest comparison is not TKGI against OpenShift but OpenShift against VKS, which keeps the platform on the VMware stack. That comparison lives in the TKGI to VKS guide, and this is the last time I will raise it, because the rest of this series assumes you picked OpenShift.

Your action on Monday: take the mapping table above, add a fifth column headed Owner, and fill in a named person for every row against your own estate. Any row where you cannot name someone is a hiring, training or partner decision, and it is far cheaper to discover that now than in Part 22 when a pilot cluster is waiting. If OpenShift AI, GPU scheduling or model serving is on your roadmap after the migration, the AI and GenAI series hub covers that ground and it will change how you size the worker MachineSets. Part 3 makes the case for why none of this can be done as an in place upgrade, whatever a vendor deck implies.

TKGI to OpenShift Series · Part 2 of 26
« Previous: Part 1  |  Guide  |  Next: Part 3 »

References

About The Author


Discover more from Journal of Intelligent Infrastructure

Subscribe to get the latest posts sent to your email.

Leave a Reply

Your email address will not be published. Required fields are marked *

Architect’s Toolkit

About the Author

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

Discover more from Journal of Intelligent Infrastructure

Subscribe now to keep reading and get access to the full archive.

Continue reading