Every TKGI plan you ever wrote is now a ResourceQuota, and every TKGI cluster you handed to a team is now a Project. That is the whole trade in this Part, and it is a better deal than it sounds. A plan bought you a fixed virtual machine shape and a BOSH deploy to change it. A quota buys you a ceiling you can raise on a Tuesday afternoon with one apply and no reboot.

A Project in OpenShift is a Kubernetes namespace with extra annotations and a lifecycle the API server owns. It is the unit of isolation, the unit of quota, and the unit of role assignment. Part 2 argued for consolidating a three cluster TKGI estate onto fewer, larger OpenShift clusters. This Part is where that argument turns into objects: the template that stamps out Projects with limits already attached, the MachineSets that supply nodes underneath them, and the autoscaler pair that grows the pool when a migration wave lands.

Preflight and Versions Tested

Prefer oc over kubectl for the rest of this series. Both talk to the same API server, but only oc knows the OpenShift specific verbs and object groups you are about to use, including oc adm create-bootstrap-project-template and the quota.openshift.io group. That is said once and not repeated.

Three things must already be true. You need cluster-admin. You need the infrastructure ID, because every MachineSet you write embeds it in four separate labels and a wrong value produces a MachineSet that reconciles into nothing. And you need to know how many compute MachineSets the installer created, because the installer provisioned ones carry a rule you will meet again in the autoscaling step.

Green here is three control plane nodes Ready, three workers Ready, and exactly one compute MachineSet named after the infrastructure ID. If oc get machinesets returns nothing, your cluster was installed user provisioned rather than installer provisioned, the Machine API has no provider to drive, and the second half of this Part does not apply to you. Part 12 covers that fork.

Plan to Project Mapping for a Three Cluster Estate

This is the artifact worth keeping from this Part. Every construct your TKGI operators use daily has a target on the OpenShift side, and in four cases out of nine the target is not the obvious one. Print this, argue about it with your platform team, then build to it.

Two of those rows deserve a decision rule rather than a mapping. Reach for a per Project ResourceQuota when the boundary you care about is one application and one team owns it, which covers most migrated TKGI workloads. Reach for a ClusterResourceQuota when a single business owner spans several Projects and you want one ceiling across all of them, because that is the shape a TKGI team estate usually had: three clusters, one budget. A ClusterResourceQuota selects Projects by label or by annotation, so decide the label taxonomy before the first wave rather than relabelling forty namespaces later. Project administrators cannot read the ClusterResourceQuota object itself, which surprises people who file tickets asking why their quota looks wrong. Point them at AppliedClusterResourceQuota instead, which is the read only view intended for them and shows the shared usage they are competing with.

One path decides whether a migrated Deployment lives or dies, and it runs in a fixed order. Defaults are injected first, quota is charged second, and admission by SecurityContextConstraints (SCC) happens last. Operators who learned Kubernetes on TKGI usually expect the SCC rejection and get blindsided by the quota one, because TKGI clusters rarely carried quotas at all.

flowchart TD
  A[oc apply Deployment] --> B[ReplicaSet creates Pod]
  B --> C{LimitRange in Project}
  C -- present --> D[Default requests and limits injected]
  C -- absent --> E[Container carries no requests]
  D --> F{ResourceQuota check}
  E --> F
  F -- within ceiling --> G{SCC evaluation}
  F -- rejected --> H[FailedCreate on ReplicaSet]
  G -- restricted v2 satisfied --> I[Pod scheduled]
  G -- denied --> J[Pod forbidden by admission]

Step by Step, Project Request Template and Default Quotas

Step 1. Generate the template. OpenShift provisions every self service Project from a template named by the projectRequestTemplate field in the cluster Project configuration. If that field is empty, the API server invents a minimal default and no quota is ever attached. Start from the built in one rather than writing yours from scratch, because the generated file already contains the RoleBindings that make the requesting user an admin of their own Project.

Step 2. Insert the ResourceQuota and the LimitRange. Both objects go into the objects list, and the definitions must sit before the parameters section of the template. The numbers below map the TKGI medium plan for the reference estate, which gave a team roughly 20 cores and 40Gi of usable request capacity.

Note the services.loadbalancers: ‘0’ line. On TKGI, NCP turned a Service of type LoadBalancer into an NSX-T virtual server automatically and teams got used to asking for them freely. On OCP with vSphere there is no such watcher, those Services sit pending forever, and a hard zero turns a confusing wait into an immediate and readable rejection. Part 13 covers what to use instead.

Step 3. Watch what happens without the LimitRange. This is the failure that costs migrations their afternoon, so build it deliberately once in a scratch Project. A ResourceQuota that names requests.cpu or requests.memory requires every incoming container to make an explicit request for those resources, and the same applies to limits. Manifests lifted off TKGI almost never carry them.

Step 4. Decide about self-provisioner, and read this before you strip it. Nearly every OpenShift hardening guide tells you to remove the self-provisioner cluster role from system:authenticated:oauth so that developers cannot create Projects. During a migration that advice is wrong, and it is wrong for a reason worth stating plainly. A migration wave creates Projects in bursts of twenty or thirty, and routing every one of them through a platform ticket queue turns a two hour cutover into a two day one. Once the quota and the LimitRange live inside the template, a self served Project is no longer an ungoverned Project. Leave the role in place until the last wave lands, then remove it if your auditors insist.

Step by Step, MachineSets and Infra Nodes on vSphere

Step 5. Build three infra MachineSets. A MachineSet is the OpenShift object that declares a pool of identically shaped nodes and asks the Machine API to keep that many alive. It is the closest thing to a TKGI plan worker definition, except it belongs to the cluster rather than to a tenant. Red Hat recommends at least three infrastructure MachineSets in a production deployment, and machines that host only infrastructure components are excluded from the total subscription count. That second sentence is the one your finance team cares about.

Step 6. Apply it and watch the first clone fail. Roughly half the vSphere MachineSets I have written failed on their first apply, and the cause was always a path in the workspace block or the template name. Red Hat Enterprise Linux CoreOS (RHCOS) machines are cloned from a vCenter template the installer left behind, and the name is not the one you guessed.

Step 7. Move the router and the registry onto them. Labelling nodes does nothing on its own. Nothing relocates until you set nodePlacement on the component itself, and if you applied the NoSchedule taint above you must also give the component a matching toleration. Miss the toleration and the router pods sit Pending while you stare at three idle infra nodes.

One more object is worth building while you are here, and most teams discover it late. A node carrying the infra label still belongs to the worker MachineConfigPool, which means every configuration change aimed at workers reboots your routers alongside your application nodes. Create a MachineConfigPool named infra whose machineConfigSelector matches both the worker and infra roles and whose nodeSelector matches the infra label. Custom pools inherit machine configs targeted at workers, so you lose nothing, and you gain the ability to drain and update infrastructure nodes on a schedule that does not collide with a migration wave. Expect one reboot when the pool is created, because the Machine Config Operator renders a fresh config for it and the member nodes apply it. Do that reboot on an empty cluster in week one, not on the morning your ingress is carrying production traffic.

The arithmetic behind this is the reason to bother. Our reference estate ran three TKGI clusters, each with its own three node control plane. Consolidating onto one OpenShift cluster deletes six control plane virtual machines outright, and moving the router, registry and monitoring stack onto tainted infra nodes takes three more machines out of the subscription count.

Step by Step, Autoscaling Without Surprise Node Churn

Step 8. Create both objects, because one alone does nothing. Autoscaling on OpenShift is two resources, not one. A ClusterAutoscaler sets cluster wide ceilings. A MachineAutoscaler binds those ceilings to a specific MachineSet. Define a ClusterAutoscaler without any MachineAutoscaler and the cluster will never scale, silently and with no error to read. That trips people who came from a platform where scaling was a single field on a plan.

Two numbers above are deliberate departures from the documented defaults. Red Hat ships unneededTime at 10m and delayAfterAdd at 10m. Both are too eager during a migration, because a Velero restore can leave a node quiet for eleven minutes while persistent volume claims bind, and a scale down in that window destroys work you cannot cheaply repeat. Thirty minutes on both costs you a handful of idle virtual machines and saves you a restarted wave. Also check that maxNodesTotal of 24 covers every machine in the cluster and not just the pool you are scaling, because it counts control plane and infra too.

Step 9. Leave the installer created MachineSets alone below their floor. Do not set minReplicas to 0 on the compute MachineSets the installer created during an installer provisioned deployment. On other platforms zero is a legitimate value and on vSphere it is accepted, which is exactly why people set it and then wonder why cluster components lost their homes.

Verification, Rollback and Failure Remediation

Run all four checks before you hand a single Project to a team. Green means a quota that shows usage rather than zeros, a ClusterResourceQuota with your Projects listed under it, three infra nodes carrying the infra role, and a MachineAutoscaler bound to a real MachineSet.

Rollback is genuinely cheap here, which is unusual in this series. Deleting a ResourceQuota releases every pending workload immediately with no restart. Reverting the project request template is a single patch that sets projectRequestTemplate back to an empty object, and existing Projects keep the quota they were born with. Only the MachineSet work needs care: scale an infra MachineSet to 0 before deleting it, confirm the router and registry pods have rescheduled onto ordinary workers, and only then remove the object. Delete a populated MachineSet and you will watch three nodes drain at once while your ingress has nowhere to land.

Sized Projects and a Node Supply That Answers

My worst afternoon with this material cost forty minutes and taught me the sequence in the diagram above. We were moving wave one of a payments estate, 22 Deployments lifted straight off a TKGI cluster. I had written a careful ResourceQuota, felt pleased about it, and shipped it in the project request template without a LimitRange. Nineteen of the 22 Deployments produced zero pods. No CrashLoopBackOff, no scheduling event, no pods at all, because the rejection happens at pod creation and lives on the ReplicaSet where nobody looks first. I spent most of that forty minutes convinced it was an SCC problem, since restricted-v2 is the usual suspect and I had been burned by it before. It was not. One LimitRange, six minutes to replay the wave, and every remaining Project inherited the fix automatically because the template carried it.

My recommendation is to treat the quota, the LimitRange, the RoleBindings and the label that a ClusterResourceQuota selects on as a single unit that only ever ships inside the project request template. Never create a Project by hand during a migration, not even a test one, because a hand made Project is the one that later fails in a way nobody can explain. Pair that with three infra MachineSets built before the first workload lands and a scale down setting you deliberately turn off for the duration of each wave. The option to avoid is the tidy looking one: locking down self-provisioner on day one and creating Projects through a request queue. It feels governed and it quietly adds a day to every wave.

On Monday, run oc adm create-bootstrap-project-template -o yaml against your own cluster and diff the generated template against whatever your team has been creating Projects with. If the two differ, you already have Projects in flight with no ceiling on them, and the fix takes an hour before it takes a week. Part 15 wires OpenShift OAuth to the same LDAP directory that UAA has been reading, and decides what happens to Harbor. If you are still weighing OpenShift against VMware vSphere Kubernetes Service as your landing place, the TKGI to VKS guide covers that alternative, and the target reference architecture in Part 11 is where the node counts in this Part came from.

References

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts