TL;DR · Key Takeaways
- A symptom is a condition. An alert definition bundles symptoms with recommendations and optional actions.
- You do not fix alert fatigue by bulk-deleting alerts after they fire. You fix it upstream.
- Wait cycles stop transient spikes from paging, and a severity ladder keeps warnings and criticals apart.
- Notification rules send only actionable alerts onward, and definitions that map to no action get deactivated.
- An alert with no recommendation and no action is noise wearing a severity label.
One host loses a power supply at 2 a.m. and your phone buzzes twenty-five times: once for the host and once for every VM riding on it. By the third buzz you have silenced the channel, and now you are flying blind for the rest of the night. That is alert fatigue, and the painful part is that the tool was technically right every single time. The failure was not detection. It was that nobody shaped detection into something a human could act on.
VCF Operations gives you the levers to fix this, but they are off the default path. The defaults are tuned to catch everything, which is the opposite of what an on-call rotation needs. This part is about turning catch everything into tell me what matters.
Symptoms, alerts, recommendations: the vocabulary
Keep two words apart and most of this gets clearer. A symptom is a single condition: as data is collected, it is compared to the symptom condition, and if the condition is true the symptom triggers. An alert definition is the thing that actually fires, and it combines one or more symptoms with recommendations that tell the responder what to do, plus optional actions the platform can run. So a symptom is a fact about one metric or property, and an alert is a curated problem statement built from facts. The recommendation is what turns an alert from a complaint into a next step.
Symptoms come in flavors that match the data model from Part 2: metric symptoms compare a number to a threshold, property symptoms check a configuration string, and there are message and fault symptoms for events. That is why the metric-versus-property distinction mattered so much earlier. The kind of symptom you can write is decided by the kind of attribute you are watching.
Four levers that cut the noise
Alert fatigue is a design problem, and these four controls are where you fix it. Use them before you ever touch the bulk-delete button.
| Lever | What it does | Use it to |
|---|---|---|
| Wait cycle | Requires a condition to persist before triggering | Kill transient spikes |
| Severity ladder | Multiple symptoms at rising severity | Separate warning from critical |
| Notification rules | Filter which alerts leave the system | Page only on actionable criticals |
| Deactivate definitions | Turns off alerts you will not act on | Cut noise at the source |
Wait cycles, the setting nobody uses
A wait cycle tells a symptom to require the condition to hold for a number of collection cycles before it triggers, and the alert definition has its own wait cycle that takes effect after the symptom has triggered. Translated to operations: a CPU spike that lasts one five-minute cycle is almost always noise, and a wait cycle of three means the condition has to persist for roughly fifteen minutes before anyone hears about it. That single setting removes the majority of flapping alerts, and it is the first thing I tune on any inherited environment.
Use a severity ladder
The docs say it plainly: use a series of symptoms to describe incremental levels of concern. Volume nearing its capacity limit is a Warning; volume reached its capacity limit is Critical. The same metric, two thresholds, two severities. This is what lets you route a Warning to a daily digest and a Critical to the pager, which is impossible if every breach is the same loud Critical. Build the ladder and the notification layer suddenly has something to filter on.
Notifications, the last gate
Inside VCF Operations every alert is visible; that is fine, because the console is for investigation. The discipline is at the boundary. Notification rules decide which alerts meet your filter criteria before they are sent to an external system, so this is where you stop the flood from reaching email, chat or a paging tool. A sane policy sends only Critical, only for production groups, and only for alert definitions that carry a real recommendation. Everything else stays in the tool for when someone goes looking. Treat the outbound notification as expensive, because a human’s attention is the scarce resource you are protecting.
Deactivate what you will not act on
Here is my contrarian position. The default set of alert definitions is not sacred, and leaving all of it active is a mistake. If an alert definition has no recommendation and maps to no action you would ever take, it is noise with a severity label, and you should deactivate it in policy. This feels wrong to administrators who equate more alerts with more safety, but the opposite is true: every dead alert you remove makes the live ones easier to trust. Coverage you ignore is worse than coverage you never had, because it trains people to swipe the channel away.
There is a planned-work version of the same idea. Maintenance generates a predictable burst of alerts, hosts entering maintenance mode, VMs moving, services bouncing, and if those reach the pager during a change window you are training people to ignore alerts at exactly the moment a real problem is most likely. Put objects into maintenance in VCF Operations before the work starts, or scope the notification rule to suppress the expected categories during the window, so the pager stays meaningful. An alert that fires during planned work and means nothing is just as corrosive to trust as a false positive, because the responder cannot tell the difference in the moment. The principle holds across all four levers: protect the channel, and people keep believing it.
Take the 2 a.m. host failure on a host with twenty-four VMs. With defaults, you can field twenty-five notifications, and the real signal, the host, is buried in the same stream as the consequences. Now apply the levers. Put a wait cycle of three on the noisy VM-level symptoms so momentary blips never escalate. Build a notification rule that only sends Critical alerts for the production group to the pager, and exclude the child VM availability symptoms that are downstream of a host being down. The result is one page: host X critical, with the affected VMs visible inside the tool when you log in. Twenty-five became one, and the one actually tells you where to start. Nothing was hidden; the consequences are still recorded, they just stopped competing with the cause for your attention.
Common objections
What is the difference between a symptom and an alert?
A symptom is a single condition that becomes true when collected data crosses it. An alert definition bundles one or more symptoms together with recommendations and optional actions. You watch symptoms; you respond to alerts.
How do I stop transient spikes from paging me?
Use wait cycles. Setting a symptom to require several collection cycles before it triggers means a brief spike never escalates, while a sustained problem still does. It is the single most effective anti-noise setting in the product.
Should I leave all the default alert definitions active?
No. Deactivate any definition that has no recommendation and maps to no action you would take. Unused alerts erode trust in the ones that matter, so a smaller, deliberate set is safer than a complete one you ignore.
How do I keep one failure from generating twenty alerts?
Use object relationships and notification filtering together. Alert on the parent, keep the downstream child symptoms as context inside the tool, and write notification rules that do not push every child symptom outbound.
How do alerts reach email, chat or a paging tool?
Through notification rules. They filter alerts by criteria such as severity and object group, then send the matching ones to the configured external system. Everything else stays inside VCF Operations.
Actions: when to let the platform fix it
An alert definition can do more than tell you about a problem. You can attach actions, so the platform runs a remediation when the alert triggers, and a recommendation can carry an action you click to run on demand. This is genuinely useful and genuinely easy to misuse, so it deserves a clear rule rather than enthusiasm.
My rule is a ladder of trust. Start with recommendations only: the alert names the fix in words, and a human decides. Graduate to on-demand actions next, where the recommendation includes a button the responder chooses to press, so the judgment stays with a person but the keystrokes are saved. Only reach fully automated actions for the narrow set of remediations that are safe, reversible and boring, the kind of thing you would let a junior do unsupervised at 3 a.m. without a second thought. Restarting a stuck management service might qualify. Anything that powers off, deletes, or moves production workloads does not, at least not until you have watched it behave for a long time.
When to let it auto-remediate
Automated remediation inherits every weakness of the alert behind it. If the symptom is noisy, you have now automated a reaction to noise, and an action that fires on a false positive can do real damage faster than a human ever could. This is exactly why the noise work comes first. Tighten the symptom with wait cycles and a sensible threshold, watch it stay quiet during normal operation, and only then consider letting it pull a trigger by itself. Recall the security point from Part 3: the vCenter account needs the rights to perform whatever you automate, so an action is also a privilege decision, not just an operations one. Automating the fix is a reward you earn after the detection is trustworthy, not a shortcut you take to avoid tuning it.
Where this leads
Alert fatigue is a design choice you can reverse. My verdict: tune wait cycles, build severity ladders, filter notifications hard, and deactivate the dead weight, all before you blame the volume on the tool. A small set of trusted alerts beats a complete set everyone mutes. Next we turn to capacity, and what running out really means, because the most useful alerts of all are the ones that warn you before you hit a wall.
By the numbers: wait cycles turn spikes into signal
Example wait cycles against the default 5-minute collection interval.
| Severity | Example wait cycles | Time to fire | Notify |
|---|---|---|---|
| Info | none | on match | Dashboard only |
| Warning | 3 cycles | 15 minutes | Email digest |
| Immediate | 2 cycles | 10 minutes | Team channel |
| Critical | 1 cycle | 5 minutes | Page on-call |
« Previous: Part 4 | VCF 9 Operations Complete Guide | Next: Part 6 »
References
- Understanding Alerts in VCF Operations, VCF 9.0 (Broadcom TechDocs)
- Symptom Definitions in VCF Operations, VCF 9.0 (Broadcom TechDocs)
- Notifications in VCF Operations, VCF 9.0 (Broadcom TechDocs)
About The Author
DrJha