TL;DR · Key Takeaways
- Performance troubleshooting is a hunt for contention, not utilization. High usage on its own is fine; waiting is the problem.
- Reach for the contention metrics first: CPU ready and co-stop, memory ballooning and swap, and disk and network latency.
- CPU ready above roughly 5 percent per vCPU is worth a look, and sustained above 10 percent is a real problem. Disk latency past 20 ms makes most applications complain.
- Oversized VMs suffer co-stop. Removing vCPUs often makes a VM faster because it stops waiting to be co-scheduled.
- Let the Troubleshooting Workbench correlate the anomaly, then walk contention first, down the stack, then sideways to the noisy neighbor.
Why utilization misleads
The single most common troubleshooting mistake is chasing high utilization. A host at 90 percent CPU that is still scheduling every VM promptly is doing its job. The signal that a workload is suffering is contention: time spent waiting for a resource it was denied. VCF Operations exposes contention directly, and those are the metrics to reach for first, before you ever look at raw usage.
For CPU, that means CPU ready (time a vCPU was runnable but waiting for a physical core) and co-stop (time a multi-vCPU VM lost because its vCPUs could not be scheduled together). For memory, it is ballooning and swap, not consumed; a VM using all its RAM is fine, a VM being swapped by the host is not. For storage it is latency, and for network it is dropped packets and latency. Usage tells you how full something is. Contention tells you who is being hurt.
There is a reason this trips up so many people. Every tool a developer or app owner has ever used reports usage, so usage feels like the truth. Virtualization adds a scheduling layer underneath the guest, and that layer is where the pain lives: a vCPU can be ready to run and still be denied a physical core for milliseconds at a time, and those milliseconds add up to a slow application even though every usage graph looks calm. The mental shift that makes an operations engineer effective is to stop asking how busy is it and start asking how long did it wait. Once contention is your first question, half the tickets that used to become arguments about hardware resolve into a placement change or a right-size, because you can point at the exact metric that proves who was waiting and why.
Symptom to likely cause
Most performance tickets map to a short list of causes. This is the table I keep in my head when a slow-VM ticket lands.
| Symptom | Metric to check | Likely cause |
|---|---|---|
| VM slow, CPU not full | CPU ready, co-stop | Host CPU overcommit; too many vCPUs |
| Sluggish app, memory looks fine | Ballooning, swap-in rate | Host memory pressure |
| Storage-heavy app stalls | Datastore / VM disk latency | Contended datastore or queue depth |
| Intermittent timeouts | Dropped packets, network latency | Uplink saturation or a bad NIC |
Let the Workbench do the correlation
The reason relationships mattered back in Part 2 is that they power correlation. The Troubleshooting Workbench takes an object and a time range and pulls together the anomalies, events and related-object behavior around the moment things went wrong. Instead of opening ten metric charts by hand, you let it surface what changed on the VM, its host, its datastore and its neighbors at the same time. That is how you separate cause from symptom: the datastore latency spike that preceded the VM slowdown is the lead, and the workbench puts them on the same timeline.
Work down, then sideways
A repeatable method beats intuition at 2 a.m. Start at the complaining object and check its contention metrics. If they are clean, move down the stack to its host and datastore, because the VM is often the victim of a busy neighbor. Then move sideways to other VMs on the same host or datastore to find the noisy one. The workbench makes this walk fast, but the discipline is yours: contention first, then down, then sideways.
Four contention lanes
Every performance problem lives in one of four lanes: CPU, memory, storage or network. Each lane has a usage metric that people watch and a contention metric that actually matters. Pin the contention metric in each lane and you have a troubleshooting cockpit that answers who is waiting, not just what is full.
Thresholds worth knowing
Numbers give the lanes teeth. CPU ready above roughly 5 percent per vCPU is worth investigating, and sustained above 10 percent is a real problem. Any host memory swap-in is bad; ballooning under mild pressure is tolerable, sustained ballooning is not. Guest disk latency in the low single-digit milliseconds is healthy for most workloads, 20 milliseconds and up is where applications start to complain, and anything above that on a latency-sensitive database is an incident. Treat these as starting points to tune to your environment, not gospel, but do not start from zero; starting from a known-good band is what lets you tell normal-busy from actually-hurting.
Build the cockpit once
Do not rebuild this under pressure. Make one troubleshooting dashboard, scoped to a cluster, that shows the four contention metrics for the top talkers plus a selectable object list, using the dashboard skills from Part 4. When a ticket lands, you open one screen, pick the VM, and read the four lanes at a glance before you ever touch the Troubleshooting Workbench for the deep correlation. The workbench is for the investigation; the cockpit is for the triage. Having both means the first sixty seconds of every performance ticket look the same, which is exactly what you want when the phone is ringing.
Scope the cockpit to a group or a cluster rather than the whole fleet, for the same reason a dashboard should never point at all objects: a triage screen that takes ten seconds to load is a screen nobody opens mid-incident. Keep it to the handful of contention metrics and one object selector, resist the urge to add pretty charts, and it stays fast. The best troubleshooting dashboard is small and unchanging, because a screen the team already knows beats a more complete one they have to relearn in the middle of an incident.
Worked example: the slow VM that was not the cause
A ticket says an app VM is slow every afternoon. Its CPU usage sits around 45 percent, so the app team blames the host. In VCF Operations you check CPU ready first: it climbs to 8 percent in the afternoon window, high enough to hurt. You go down to the host: CPU usage is 92 percent from 2 to 4 pm. You go sideways: a reporting VM on the same host runs a heavy batch job in exactly that window, pinning the cores. The complaining VM was never the cause; it was the victim of a neighbor. The fix is not more hardware. It is moving or rescheduling the batch VM, or letting DRS separate them. Utilization pointed at the wrong VM. Contention plus the neighbor walk pointed at the right one, in minutes.
Common objections
The host is at 90 percent, surely that is the problem?
Not necessarily. High utilization with low contention means the host is working efficiently. Only when ready, co-stop, ballooning or latency climb are workloads actually being hurt. Check contention before you act on a usage number.
Isn’t giving a slow VM more vCPUs the fix?
Often it makes things worse. More vCPUs raise co-stop because the scheduler must co-schedule them all. Size vCPUs to what the workload uses; for many VMs, fewer vCPUs run faster.
What is the Troubleshooting Workbench for?
It correlates an object’s anomalies, events and related-object behavior over a time range, so you see what changed across the VM, host, datastore and neighbors together instead of opening charts one by one.
Where do I even start with a vague slow ticket?
At the complaining object’s contention metrics, then move down to its host and datastore, then sideways to co-located VMs. That fixed order finds the noisy neighbor fast and keeps you from guessing.
Patterns that waste hours
A few habits turn a ten-minute diagnosis into a half-day. The first is rebooting the VM. It clears the ticket for an hour and destroys the evidence, so when the slowness returns you are starting from nothing. Before any reboot, capture the time window and let the workbench snapshot the correlation; the reboot can wait ninety seconds.
The second is trusting in-guest tools alone. Task Manager or top inside the VM cannot see co-stop, ballooning or the busy neighbor on the same host, because those happen at the hypervisor layer the guest is blind to. The guest says the CPU is idle while the VM is starved of physical cycles. This is exactly why VCF Operations exists: it sees the layer the guest cannot. When the guest and the platform disagree, the platform is telling you the truth about scheduling.
The third is averaging. A daily average hides the afternoon spike that is the whole problem. Always narrow to the window the user complained about and read the peak, not the mean. A VM that averages 3 percent CPU ready but hits 15 percent for two hours every afternoon is failing for those two hours, and the average will talk you out of believing the user. Read the peak inside the window the user complained about, not the daily average.
When it really is capacity
Sometimes the neighbor walk ends with every VM on the host behaving and the host itself simply out of room: contention is high across the board, no single workload is abusive, and DRS has nowhere better to place anyone. That is the honest case for more hardware, and it is also where this part hands off to capacity work. Performance troubleshooting and capacity planning are the same conversation seen from two ends; a cluster that is chronically contended is a cluster that is chronically short, and the fix is the capacity discipline from Part 6, not another afternoon of chart-reading. The skill is telling the two apart: a noisy neighbor is a placement problem you fix for free, a saturated cluster is a capacity problem you plan and buy for.
Where this leads
Performance work is contention work. Read who is waiting, let the workbench correlate, and walk down then sideways to the cause. My verdict: ban utilization-only troubleshooting, standardize the walk, and right-size before you buy. With monitoring, capacity and troubleshooting in hand, the next phase turns to money, starting with cost visibility and showback.
One habit ties this part to everything before it. When you close a performance ticket, write down which lane it was, the metric that proved it, and the fix. Over a quarter those notes become your environment’s real thresholds and your real recurring causes, and they turn the next engineer’s guesswork into a lookup. The teams that troubleshoot fast are not the ones with the sharpest intuition; they are the ones who wrote down what the last ten tickets actually were, so the next similar ticket becomes a lookup instead of a fresh investigation. Contention metrics find the cause today; a short, honest ticket history is what makes tomorrow faster.
By the numbers: contention thresholds to keep in your head
Starting bands to tune to your environment, not gospel.
| Metric | Healthy | Investigate | Incident |
|---|---|---|---|
| CPU ready, per vCPU | under 5 percent | 5 to 10 percent | sustained above 10 percent |
| CPU co-stop | near zero | any sustained | high and rising |
| Memory ballooning | none | mild under pressure | sustained |
| Memory swap-in | none | any | sustained |
| Guest disk latency | low single-digit ms | around 20 ms | well above 20 ms |
« Previous: Part 6 | VCF 9 Operations Complete Guide | Next: Part 8 »
References
- Troubleshooting Using the Troubleshooting Workbench, VCF 9.0 (Broadcom TechDocs)
- Viewing Metrics and Properties, VCF 9.0 (Broadcom TechDocs)
- Using VCF Operations Diagnostics, VCF 9.0 (Broadcom TechDocs)
About The Author
DrJha