← Back to the GPU series: all 11 demos

GPU Autoscaling Lab

A day of chatbot traffic · vLLM on up to 8 GPUs · four ways to scale
00:00

A chatbot runs on vLLM, one GPU per replica, up to 8 replicas. Each replica serves about 20 requests a second before latency climbs. Traffic follows a normal working day with an evening peak; you can add a sudden spike. Pick a scaling policy and watch a whole day run in about a minute. Numbers are illustrative; the behaviour matches the real tools.

Scaling policy

Settings

Last 24 hours

requests/secondcapacity (ready replicas × 20)p95 latency (right axis)

Replicas right now

Config

What happened