TL;DR · Key Takeaways
- The worst anti-pattern is forwarding every alert to a ticket queue or inbox. The noise gets filtered, and the one real alert sits unread while a customer finds the problem first.
- Policy sprawl is next. Most environments need one or two policies, not fourteen. More policies means more places for a setting to hide.
- Treating out-of-the-box thresholds as truth creates most of the noise. The default datastore latency tolerance around 20 to 30 ms fires constantly on arrays that blip harmlessly.
- On one site, tuning took alerts from about 1,200 a week at 8 percent actionable to about 90 a week at 88 percent actionable.
- Do not delete built-in alerts. Disable them in the policy or clone and modify, so you can always see the default again.
- Every anti-pattern in this part has a fix that costs an afternoon and saves a quarter of firefighting.
Every environment I inherit has the same handful of self-inflicted wounds. Nobody set out to build them. They accreted, one reasonable-looking decision at a time, until the tool that was supposed to give clarity was producing noise. The good news is that anti-patterns are predictable, and predictable means fixable.
This part is the field guide to the ones I see most, why each feels sensible when you build it, and the specific fix. I will go deepest on alert fatigue, because it is the one that hurts most and the one with the clearest numbers, and then run through the rest with the same before-and-after honesty.
Ranking them by how much they hurt
Not all anti-patterns are equal. Some cost you a slow console, some cost you an outage. Before the detail, here is how I rank the common ones by blast radius, because that is the order you should fix them in. The loud, low-severity habits like dashboard hoarding can wait. The quiet, high-severity ones, forwarding everything and running a single proxy, are the ones that let a real problem through.
Alert fatigue, the expensive one
The most common and most costly anti-pattern is forwarding everything. Every alert goes to the ticket queue or an inbox, on the theory that catching everything is safer. The opposite happens. The volume is too high to read, so it gets filtered to a folder, and the one alert that mattered, a datastore filling up, sits in a sea of harmless latency notices. The failure mode is rarely a missed alert. More often it is a hundred meaningless alerts burying the one that counts, until a customer reports the outage you were paged about three hours earlier.
Most of that noise comes from the second anti-pattern hiding inside it: treating the out-of-the-box thresholds as truth. The defaults are global best practices, not your environment. The vSphere pack ships a datastore latency tolerance around 20 to 30 milliseconds, which is sensible in general and wrong for an all-flash array that normally runs at 5 to 8 and blips to 22 without anyone caring. So the alert fires constantly, everyone learns to ignore it, and the ignoring spreads to alerts that do matter.
The cleanup, with numbers
One site had inherited 14 policies and every default alert enabled, producing roughly 1,200 alerts a week. When we measured how many were actionable, meaning a human did something in response, it was about 8 percent. The rest were noise, and the latency alert above was the loudest source. We did three things. We consolidated to 2 policies, both inheriting from the default so we could still see what the default said. We disabled the noisy built-in alerts in the policy rather than deleting them, so they were recoverable. And we retuned latency to fire on sustained readings above 30 rather than momentary blips over 20. Alert volume fell to about 90 a week, and the actionable share rose to about 88 percent.
The rest of the field guide
The other anti-patterns each have the same structure: a reasonable instinct, a bad outcome, and a cheap fix. The table pairs the symptom you will notice with the fix I would apply, and the before-and-after table after it puts numbers on what a full cleanup looks like across an environment.
| Anti-pattern | What you notice | Fix |
|---|---|---|
| Forward everything | Ticket queue nobody reads | Forward only actionable alerts |
| Defaults as truth | Constant latency and low-severity noise | Tune thresholds to your hardware |
| Policy sprawl | Settings hiding in one of fourteen policies | Consolidate, inherit from default |
| Watching utilization | Busy alerts that mean nothing | Chase contention, not utilization |
| Super metric sprawl | Slow collection, duplicate formulas | Delete duplicates of built-ins |
| Single proxy per site | A site goes blind on failure | Collector group of two or more |
| Measure | Before | After |
|---|---|---|
| Alerts per week | ~1,200 | ~90 |
| Actionable share | 8% | 88% |
| Policies | 14 | 2 |
| Custom dashboards | 40 | 9 |
| Proxies at busy site | 1 | 2 in a group |
Three more worth naming
Policy sprawl
Policies feel like control, so teams keep making them. One for production, one for test, one for that one cluster, one for the project that ended two years ago. Before long a setting you are looking for could live in any of fourteen places, and changing a threshold means hunting through all of them to find which policy actually applies to the object in front of you. The fix is boring and effective: collapse down to one or two policies, both inheriting from the default. Inheritance is the trick people miss. When your policy inherits from the default, you can always see what the default said and what you changed, which turns a policy from a black box into a diff.
Chasing utilization instead of contention
A VM at 90 percent CPU is not a problem. A VM waiting on CPU it cannot get is a problem, and those are different numbers. The anti-pattern is alerting on utilization, which is high by design on a well-packed host, and then chasing busy VMs that are working exactly as intended. You generate noise and you miss the actual contention, the ready and co-stop time that means a VM is being starved. I covered the mechanics in the performance part of this series, and the anti-pattern version of it is simply forgetting that lesson and wiring an alert to utilization because it is the number that is easy to see.
Hoarding dashboards and super metrics
These two travel together. Nobody deletes a dashboard, because deleting feels risky, so forty accumulate and two get opened. Nobody checks whether a super metric duplicates a built-in, so thirty pile up and half recompute numbers the platform already collects, taxing every cycle. Both have the same cure: measure use and cut. Rank dashboards by opens and keep the ones with traffic and an owner. Map each super metric to a question and delete the ones that answer none. The clutter is not neutral. It slows the console, it slows collection, and it hides the things that matter behind the things that do not.
Why anti-patterns keep coming back
Fixing an anti-pattern once is easy. Keeping it fixed is the hard part, because the instincts that create them are reasonable. Forwarding everything feels responsible. More policies feel like more control. Keeping every dashboard feels safe. The drift back happens quietly, a policy added here for one team, a dashboard cloned there and never deleted, until you are back where you started. The defense is the runbook from the previous part. The weekly and monthly reviews are where you catch a fourteenth policy before it becomes normal, and the quarterly review is where you re-measure the actionable share and the dashboard count and prune what crept back.
The deeper fix is cultural. An environment stays clean when the team agrees that a new policy needs a reason, a new dashboard needs an owner, and a new forwarded alert needs to be actionable. None of that is a product feature. It is a habit, and the habit is what the maturity model in the final part is really about.
Fix them in the right order
When you inherit a messy environment, the instinct is to start with the thing that annoys you most, which is usually the cluttered dashboards because they are what you stare at. That is backwards. Fix in order of blast radius, top of the severity chart first. Sort out alert routing so only actionable alerts leave the console, because that is the one that lets a real problem through. Then put a second proxy at any site running on one, because that is the other silent, high-severity gap. Only after the dangerous ones are handled do you get to the satisfying cleanup of policies, super metrics and dashboards.
There is a reason for the order beyond severity. The high-severity fixes are also the ones that make the rest easier to see. Once the noise is gone, the genuinely useful dashboards are obvious and the useless ones are exposed. Once each site has a healthy collector group, the completeness signal you would use to judge everything else is trustworthy. Clean the signal first, then clean what the signal reveals. Doing it the other way round means tidying dashboards you will end up deleting anyway once the alerting makes clear which ones nobody needed.
One more caution. Do the cleanup in daylight and in small steps, not as one heroic weekend rewrite. Disable a batch of noisy alerts, watch a week, measure the actionable share, then do the next batch. A big-bang cleanup risks silencing something you needed, and it gives you no clean before-and-after to learn from. The whole point of measuring the actionable share is lost if you change forty things at once and cannot tell which one helped.
Common questions
Should I ever forward alerts to ticketing?
Yes, but only the actionable ones. Route the alerts a human will act on to the queue and leave the rest in the console for triage. The queue should be a to-do list, not a firehose.
How many policies should I have?
One or two for most environments. If you have more than a handful, you are probably hiding settings from yourself. Consolidate and inherit from the default so you can always see what changed.
Can I just delete the noisy built-in alerts?
Do not delete them. Disable them in the policy, or clone and modify the clone and disable the original. Deleting means you lose the default and cannot compare against it later.
How do I know if I have an alert fatigue problem?
Measure the actionable share. If far fewer than half of the alerts that fire lead to any action, you have alert fatigue, and the tuning will pay for itself in a week.
Are the defaults just wrong?
No, they are general. They are built for a typical environment and are a fine starting point. They become an anti-pattern only when you leave them unexamined on hardware they do not fit.
I inherited a mess. Where do I start?
With alert routing, not dashboards. Stop forwarding non-actionable alerts, then add a second proxy anywhere a site runs on one. Those two fixes close the gaps that let real problems through. The dashboard and policy cleanup is satisfying but lower risk, so it comes after.
Should I clean up in one big push?
No. Change a batch, watch a week, measure the actionable share, then continue. A big-bang cleanup can silence something you needed and leaves you no clean before-and-after to learn from. Small steps in daylight beat a heroic weekend.
« Previous: Part 16 | VCF 9 Operations Complete Guide | Next: Part 18 »
References
- VCF Operations: Configuring Alerts and Using Actions
- VCF Operations: Configuring and Managing Policies
- Operations Policies: What, Why, How
About The Author
DrJha