Two numbers that decide every scaling action

Thirty eight and seventy one. On the morning I started drafting this Part, aggregate CPU requests across the six worker nodes in the ns-payments cluster of our reference estate sat at 38 percent of allocatable, while measured CPU consumption over the same fifteen minute window sat at 71 percent. Both figures are correct. Only one of them is visible to anything that scales. Kubernetes places pods on requests. Cluster Autoscaler reacts to pods that cannot be placed, which is a request question. HPA reacts to utilisation expressed as a percentage of requests. A 33 point gap between what the workloads asked for and what they take means every automatic mechanism in that cluster is reasoning from a figure that bears no relation to reality.

That single observation is most of objective 5.5. Optimisation on VKS is a measurement you repair first and a lever you pull second, and getting those two in the wrong order produces clusters that grow nodes forever while individual pods stay slow. Popular study advice for this objective is to learn the autoscaler flags. Learn them, certainly, but understand that a cluster with unset or badly guessed requests will make the autoscaler behave like a random number generator with a credit card.

Trace the hard ceiling back to the Supervisor, because that is where it lives and this series has said so in every Part that touched a workload cluster. A VKS worker node is a virtual machine sized by a VM Class, and a VM Class becomes usable only when a vSphere administrator binds it to the vSphere Namespace that owns the cluster. If best-effort-large at 4 vCPU and 16 GB is the only class bound to ns-payments, a pod requesting 6 CPU will sit Pending indefinitely while Cluster Autoscaler cheerfully adds nodes that pod can never fit on, right up to the maximum you configured. Horizontal scaling cannot change the shape of a node. Every scaling conversation in VKS eventually terminates at a Supervisor object, and on this objective it terminates at a VM Class or at the vSphere Namespace CPU and memory limits.

Worth restating once for this final Part: VKS is the product formerly called TKG Service or TKGS. Broadcom renamed the product and left the plumbing alone, so scaling documentation still sits under URL paths containing operating-tkg-service-clusters and autoscaling-tkg-service-clusters, node pools are Cluster API MachineDeployments, and autoscaler annotations carry the cluster.x-k8s.io prefix. None of that is stale documentation. It is what your estate actually calls things.

Preflight for a measurement pass

Last Part we rescued an upgrade that had wedged on a drain. This Part the cluster is healthy, every Machine is Running on the target release, and the complaint is latency rather than failure. Two contexts still apply and you will move between them constantly. Supervisor context holds the Cluster, MachineDeployment, VM Class binding and vSphere Namespace limits. Workload cluster context holds the pods, the metrics API, HPA objects and the Prometheus deployment. I use kubectl with the vSphere plugin for both, because the plugin issues one credential that covers Supervisor and workload cluster contexts and swapping with kubectl config use-context is faster than juggling kubeconfig files, which matters when you are timing a scale event.

Establish versions before anything else, because autoscaler behaviour and package availability both move with them, and because a mismatched package minor version fails reconciliation silently enough to waste an afternoon.

If that command returns nothing useful, you have found your first problem rather than a broken tool. Here is the real failure line, and it is the single most common blocker on this objective because everything downstream of it depends on the resource metrics API being served.

Read that carefully, because it is a small joke at your expense. Your monitoring add on cannot be scheduled for exactly the reason you were about to investigate. Insufficient cpu here is a request arithmetic statement, not a busy CPU statement, and the fix is either honest requests on the workloads squatting on the reservation or one more node. That is also a useful reminder that the resource metrics API is not optional infrastructure on this objective, since HPA and kubectl top both read from it.

Metrics pipeline from kubelet to dashboard

Three distinct layers exist on a VCF 9.0 estate and candidates routinely treat them as one thing. Layer one is the resource metrics pipeline: kubelet exposes CPU and memory, metrics-server aggregates it, and the API server serves it at metrics.k8s.io. Upstream Kubernetes documentation is unambiguous that this layer exists for autoscaling and for kubectl top, and it explicitly warns against using metrics-server as a monitoring solution. It keeps a short in memory window and no history at all. Layer two is Prometheus, delivered as a VKS standard package, which scrapes and retains, with Grafana on top for visualisation. Layer three is VCF Operations, which correlates Kubernetes objects with the ESXi hosts, datastores and vSAN underneath them, and which is the only layer that will tell you a pod is slow because a disk group is rebuilding.

Here is where this objective quietly reaches back into Part 7. Broadcom documents that the Prometheus and Grafana packages require a default storage class on the VKS cluster where you deploy them. A default storage class exists only because a vSphere Storage Policy was assigned to the vSphere Namespace on the Supervisor and then nominated as default in the cluster spec. Skip that and your monitoring stack never comes up, which is a Supervisor failure presenting as a monitoring failure. It is a neat illustration of the third premise of this whole series.

Fix that on the Supervisor by assigning the storage policy to the vSphere Namespace, then nominate it as the default in the cluster spec, and the package reconciles on its own within a scrape cycle. Do not annotate a storage class as default by hand inside the workload cluster as a shortcut, because the next cluster reconciliation from the Supervisor topology will quietly undo it and your monitoring will fall over on a Tuesday for no visible reason.

flowchart TD
  A[Workload reported slow] --> B[Compare requests against actual usage]
  B --> C{Requests set and realistic}
  C -->|No| D[Correct requests and limits first]
  D --> B
  C -->|Yes| E{Any pods Pending}
  E -->|Yes| F[Read the scheduling event text]
  F --> G{Insufficient cpu or memory}
  G -->|Yes| H[Add worker nodes or raise namespace limits]
  G -->|No| I[Check VM class binding and node selectors]
  E -->|No| J{Utilisation above HPA target}
  J -->|Yes| K[Scale pods horizontally]
  J -->|No| L[Investigate storage latency and network path]
  H --> M[Measure again before the next change]
  I --> M
  K --> M
  L --> M

Scaling levers and what each one actually moves

Keep this table. It is the artifact from this Part worth returning to, and it is built around the question the exam keeps asking in different costumes, which is not how do I scale but which thing do I scale. Column four is the one that saves you, because most wasted optimisation effort is a correct lever applied to the wrong bottleneck.

Horizontal pod scaling comes first because it is cheapest and fastest. Note the requests block in this manifest, since without it the HPA has no denominator and reports unknown forever, which is the failure I have seen more than any other on this objective.

Add a CPU request to that container and the same HPA reports a real percentage within one sync interval, which defaults to fifteen seconds. Node level scaling is the next lever, and Part 22 covered the install in full, so here it is only the annotation surface that decides behaviour, applied on the Supervisor against the Cluster topology.

Verification, rollback and failure signatures

Green here is not a running pod, it is a closed loop. Four checks prove that loop: metrics are being served, every HPA has a numeric current value, the node pool reports its replicas ready and updated in equal numbers, and the vSphere Namespace still has headroom against its configured limits. Miss the fourth and you will discover that your autoscaler maximum was fiction all along.

Rollback on this objective is pleasantly boring, which is a good argument for measurement driven changes over configuration sprawl. Remove a HPA and the Deployment holds whatever replica count it last had, so scale it back explicitly rather than assuming. Remove the autoscaler annotations and the pool freezes at its current replica count, which is safe. Reverse a manual MachineDeployment scale by setting the old number, and expect a drain on the way down. A VM Class change is the only lever with no cheap reverse, because reverting it rolls every node again, which is why it belongs in a change window and never in an incident.

Exam focus for objective 5.5

Objective checkpoint

Optimization order I would ship for this estate

Here is the war story that produced that table. A retail customer opened a severity two on a Thursday afternoon in November because checkout latency had tripled. Their platform team had already scaled the node pool from six to twelve over two days, which cost roughly 96 vCPU and 384 GB of committed namespace quota and moved p95 latency by 40 milliseconds in the wrong direction. I spent nineteen minutes on the numbers before touching anything and found aggregate requests at 31 percent of allocatable, a checkout Deployment with no CPU request at all, and a HPA that had been sitting at unknown over 70 for eleven days without anybody noticing. One request block on one container, one HPA resync, and that Deployment scaled itself from four replicas to fifteen inside two minutes. We took the node pool back to seven the following week. Total repair time once I stopped guessing was under half an hour, against two days of scaling that had made things marginally worse.

Verdict, and it argues with the most common advice on this objective: enable Cluster Autoscaler last, not first. Popular guidance treats the autoscaler as the answer to performance because it is the most visible feature in the chapter, and on a cluster with honest resource requests it is genuinely good. On a cluster without them it converts a workload problem into an infrastructure bill and hides the evidence while doing it. Order I would ship, and the order I would reason through an exam item in: repair the metrics pipeline, set requests that match observed usage, apply HPA, then and only then bound the node pool with an autoscaler, and treat VM Class changes as a planned design decision rather than a response to an alert. Avoid the pattern of raising the autoscaler maximum during an incident, because it always works for twenty minutes and always costs more than it saved.

Clean result on this objective looks like five things: kubectl top returns numbers for every node, every HPA in the cluster reports a numeric current value rather than unknown, MachineDeployment READY equals REPLICAS equals UPDATED, aggregate requests sit within about 15 points of measured usage, and the vSphere Namespace has visible headroom above the autoscaler maximum so that the ceiling you configured is the ceiling you get. Do this tonight in your own lab: deploy the stateless web Deployment with no CPU request, attach a HPA, watch it report unknown, then add the request and time how long it takes to scale. Then set the autoscaler maximum higher than the namespace CPU limit and generate load, so you see for yourself exactly which object refuses first. Component depth on every piece named here lives in the vSphere Kubernetes Service complete guide, autoscaler install and version pinning is in Part 22, and the lifecycle commands you will lean on while measuring are in Part 20. That closes the series. Thirty four Parts, one published objective each after the orientation, and the whole path is indexed on the VCAP-VKS exam guide. Go and book the exam.

References

• About Cluster Autoscaling, Broadcom TechDocs, vSphere Supervisor 9.0
• Manually Scale a Cluster Using Kubectl, Broadcom TechDocs, vSphere Supervisor 9.0
• VKS Standard Packages Release Notes, Broadcom TechDocs
• Resource metrics pipeline, Kubernetes documentation
• Horizontal Pod Autoscaling, Kubernetes documentation
• VMware Cloud Foundation VKS Administrator Exam Guide, 3V0-24.25

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts