TL;DR · Key Takeaways
- The Cluster Autoscaler adds and removes worker nodes based on pending pods and idle capacity. The Horizontal Pod Autoscaler scales pods. Run both, tuned to cooperate.
- You install the autoscaler into the cluster (VCF CLI or kubectl) and set per-node-pool min and max instead of hard-coding replicas.
- Scale-up is triggered by unschedulable pods; scale-down removes underused nodes after a cool-down, and it respects pod disruption budgets.
- The autoscaler can only act within your namespace quota and host capacity. An ambitious max against a tight quota just produces stuck pods, the quota always wins.
Static clusters waste money at the trough and fall over at the peak. The Cluster Autoscaler is how you stop guessing node counts and let demand drive them. On VKS it works the way it does on any Cluster API platform, adding and removing worker VMs, but with VKS-specific guardrails you must respect: the namespace quota and the VM classes from Part 5 bound what it is allowed to do. Understand those bounds and the autoscaler is reliable; ignore them and you get the “autoscaler is broken” ticket that is really a quota problem.
Two autoscalers, different jobs
It is worth separating the two things people lump together. The Horizontal Pod Autoscaler (HPA) changes how many pod replicas a workload runs, based on CPU, memory or custom metrics. The Cluster Autoscaler (CA) changes how many nodes the cluster has, based on whether pods can be scheduled. They are complementary: the HPA creates more pods, and if those pods cannot fit, the CA adds nodes for them; when load drops, the HPA removes pods and the CA reclaims the now-idle nodes. Tune them so they cooperate rather than fight, an HPA that flaps will make the CA churn nodes expensively.
Setting min and max, and what they really mean
On VKS you install the Cluster Autoscaler into the workload cluster, the docs cover both a VCF CLI install and a kubectl install, and configure it against the node pools. The key shift is that you no longer hard-code the worker replica count; you give each pool a minimum and a maximum, and Kubernetes manages the actual number between them. Set the minimum to the steady-state load you always expect, so the cluster is never caught flat-footed, and set the maximum to what your namespace quota and host capacity can genuinely satisfy. A maximum the quota cannot meet is worse than useless: the autoscaler tries to scale up, fails to get resources, and your pending pods stay pending.
| Behaviour | Trigger | What to watch |
|---|---|---|
| Scale up | Unschedulable (pending) pods | New-node provisioning time; quota headroom |
| Scale down | Nodes underused past a cool-down | Pod disruption budgets; pods that block drain |
| No-op (stuck) | Max reached or quota exhausted | Raise max only if quota and capacity allow |
What bites in production
Three things catch people. First, scale-up is not instant: a new node is a new VM that must be provisioned, booted and joined, so latency-sensitive bursts still need a sensible minimum rather than relying on cold scale-up. Second, scale-down is polite: it respects pod disruption budgets and refuses to drain a node if doing so would violate one, so a single mis-set PDB or an unevictable pod can pin nodes you wanted reclaimed. Third, the autoscaler and your namespace quota must agree, and the quota always wins. The most common “autoscaler is broken” report is really “the quota was exhausted and nobody told the autoscaler it could not have more.”
How the autoscaler chooses which pool to grow
When you have more than one node pool, the autoscaler does not pick at random; it grows the pool whose nodes can actually satisfy the pending pods. If a pod requests a GPU, only the GPU pool can host it, so that is the pool that scales. If a pod just needs CPU and memory, the autoscaler weighs the candidate pools and, with the default expander, grows the one that wastes the least capacity for the pending workload. This matters because it gives you a lever: by shaping pods with the right resource requests, node selectors and taints, you steer which pool absorbs which growth. Tag your expensive pools (GPU, high-memory) so that only the workloads that truly need them can trigger their growth, or you will watch a mislabelled batch job scale up a pool of GPU nodes nobody intended to pay for.
The corollary is that resource requests are not a formality. The autoscaler reasons entirely from requests, not actual usage, so a pod that requests far more than it uses makes the cluster scale up for capacity it never consumes, and a pod that requests too little gets packed onto a node and then starved. Right-sizing requests is half of making autoscaling behave, and it is the half developers most often skip.
Tuning scale-down so it actually reclaims
Scale-up tends to just work; scale-down is where the autoscaler quietly underperforms, and almost always for the same reasons. A node will only be removed when its pods can be rescheduled elsewhere and no pod disruption budget blocks the eviction. So a single replica with a strict PDB, a pod with local storage the scheduler will not move, or a system daemon without a tolerant budget can pin an otherwise idle node indefinitely. The cluster looks like the autoscaler is broken; really it is doing exactly what it was told and refusing to violate a constraint you set. The fix is to audit pod disruption budgets and make sure batch and stateless workloads can actually be evicted, and to give system add-ons sane budgets rather than implicit ones that block drains.
The scale-down delay and utilisation threshold are also worth tuning to your workload’s rhythm. Set them too aggressive and the cluster thrashes, adding and removing nodes as load wobbles, which on VKS means constantly provisioning and destroying VMs. Set them too conservative and you pay for idle nodes long after the peak passed. Match the timing to how spiky your real traffic is, and remember that on VKS a node is a VM with a real provisioning cost, so a little hysteresis is cheaper than churn.
Pairing it with the HPA, and the metrics it needs
The Cluster Autoscaler only does half the job. It adds nodes when pods cannot fit, but it does not create the pods, that is the Horizontal Pod Autoscaler’s role, and the two have to be wired together to get real elasticity. The HPA scales replicas based on metrics, so it needs a working metrics pipeline: the metrics server for basic CPU and memory, or a custom and external metrics adapter (often fed from Prometheus) for anything more interesting like queue depth or requests per second. Without that pipeline the HPA has nothing to act on, and the Cluster Autoscaler then has no surge of pending pods to respond to, so the cluster sits at its floor while the workload struggles.
So the chain you actually want is: real metrics feed the HPA, the HPA adds pods, pods that do not fit make the Cluster Autoscaler add nodes, and the whole thing reverses cleanly as load falls. Tune the HPA so it does not flap, because a flapping HPA makes the Cluster Autoscaler churn VMs, which is the expensive failure mode. Scale on a metric that reflects actual demand rather than a lagging proxy, give the HPA a stabilisation window, and the node-level and pod-level autoscalers will cooperate instead of fighting.
Why scale-down stalls: a failure map
Scale-up is rarely the problem; scale-down is where the autoscaler quietly underperforms, and it is almost always one of three blockers. A node will only be removed when every pod on it can move elsewhere, so a strict pod disruption budget, an unevictable pod, or a pod with local storage can each pin an idle node indefinitely. The cluster looks like the autoscaler is broken when it is faithfully refusing to violate a constraint you set.
Making nodes reclaimable
The fix is to audit what makes nodes un-drainable. Give batch and stateless workloads pod disruption budgets that actually allow eviction, give system add-ons sane budgets rather than implicit blocking ones, and keep local-storage pods on a pool you do not expect to scale down. Once nodes can drain, scale-down works, and the cluster stops paying for idle capacity it could not reclaim.
In practice
When a cluster will not shrink after a peak, I do not touch the autoscaler config first. I list the pods on the idle nodes and ask which one cannot move. It is nearly always a single-replica workload with a one-of-one disruption budget, or a logging or monitoring daemon with no eviction tolerance. Fix the budget, and the node the autoscaler has been politely refusing to drain disappears within minutes.
Requests drive everything
The autoscaler reasons entirely from resource requests, never from actual usage, and that single fact explains most autoscaling surprises. A pod that requests far more than it uses makes the cluster scale up for capacity nobody consumes; a pod that requests too little gets packed onto a node and then starved. Picture the gap between requested and used and the waste becomes obvious.
Steering growth onto the right pool
Because requests and selectors decide where pods land, they also decide which pool the autoscaler grows. Tag expensive pools, GPU, high-memory, so only workloads that genuinely need them can trigger their growth, or a mislabelled batch job will scale up a pool of GPU nodes nobody intended to pay for. Right-sized requests plus deliberate node selection keep both the cost and the placement under control.
Worked example
A team copied a Helm chart that requested 4 CPU per pod for a service that used a quarter of that at peak. The autoscaler dutifully added nodes to fund the requested capacity, the cluster grew, and the cloud-style bill grew with it, all for CPU that sat idle. Trimming the requests to match real usage shrank the cluster by a third overnight with no impact on performance. The autoscaler had been doing exactly what the requests told it; the requests were the bug.
Scale-to-zero and the cold-start cost
For genuinely intermittent workloads, a node pool can have a minimum of zero, so it costs nothing when idle and grows only when work arrives. That is attractive for batch jobs and bursty, infrequent services, but it comes with a cold-start cost that has to be acceptable: when the first pod arrives, there is no node, so the autoscaler must provision a VM, boot it and join it before anything runs, which is minutes, not milliseconds. For a nightly batch job that latency is invisible; for a user-facing request it is a timeout. Use scale-to-zero where the workload can tolerate the wait, and keep a minimum of one or more where latency matters so there is always a warm node ready. The choice is a direct trade between cost at idle and responsiveness at the start of a burst, and it should be made per pool against the workload’s tolerance, not applied as a blanket policy.
What I’d Do
I run the Cluster Autoscaler and the HPA together on anything with variable load, and I tune the HPA’s thresholds so it does not flap and make the CA churn VMs. I set the node-pool minimum to real steady-state so bursts are not waiting on a cold node boot, and I set the maximum only after checking it against the namespace quota and host headroom, never as an aspiration. I audit pod disruption budgets so scale-down is not silently pinned by a single-replica workload. Done well, autoscaling turns fixed pools into elastic ones and nobody notices it working, which is the goal. Looking at your busiest cluster: if it hit its autoscaler maximum tomorrow, would the quota actually let it grow, or would the pods just queue?
References
- Broadcom TechDocs: Install Cluster Autoscaler by Using VCF CLI
- Broadcom TechDocs: Install Cluster Autoscaler Using Kubectl
- 5 Things That Break VKS Clusters in VCF 9, drpranayjha.com
About The Author
DrJha