TL;DR · Key Takeaways
- A runbook is a cadence, not a document. Daily, weekly, monthly and quarterly checks, each tied to a real signal, so drift gets caught while it is still small.
- Daily takes about ten minutes: triage overnight alerts, confirm collection completeness is 100 percent, check the platform health endpoint, and verify last night’s backup wrote to SFTP.
- Watch the hard thresholds. The platform health check fails a node when any disk breaches 90 percent, expects no VM snapshots on nodes, and needs backups landing on the SFTP target.
- The health endpoint lives on the primary VIP on port 30006. If you are not looking at it, a failing check is invisible until something stalls.
- On one site, the daily backup check caught a job that had silently stopped writing to SFTP. Because it was a daily item, the gap was one day, not discovered at the next restore.
- Automate the checks you can, keep the human on the trends. A runbook nobody runs is just a file.
Most outages I have seen in operations tooling were not sudden. A disk crept up over three weeks. A backup stopped writing and nobody looked. An alert got noisy and got muted and then the real one hid in the noise. None of that needs to happen, and the thing that prevents it is dull: a runbook you actually run, on a cadence, tied to signals rather than vibes.
This part turns the rest of the series into a repeatable practice. I will lay out the daily, weekly, monthly and quarterly checks I run for VCF Operations itself, the thresholds that matter, and a real case where a ten-minute daily habit caught a backup that had quietly failed.
The cadence, not the document
A runbook written once and filed is worthless within a quarter. What works is a short list of checks bound to a rhythm, where each item exists because skipping it has burned someone. The point of the cadence is that different problems move at different speeds. Alert noise and backups need a daily eye. Capacity trends and cost want a monthly look. Sizing and recovery are quarterly. Match the check to the speed of the thing it watches, and you spend a little time often instead of a lot of time after an incident.
| Cadence | Focus | Core checks |
|---|---|---|
| Daily | Is it healthy right now | Alerts, completeness, health endpoint, backup |
| Weekly | Is it staying tidy | Alert tuning, reclamation, snapshots, drift |
| Monthly | Is it trending well | Capacity forecast, cost, policy, dashboard prune |
| Quarterly | Is it built for what’s next | Sizing, recovery test, audits, upgrade plan |
The daily ten minutes
The daily pass is the one that earns its keep, and it should take about ten minutes. Four things. Triage the overnight alerts, separating the ones that mean something changed from the noise. Confirm collection completeness is at 100 percent, because a proxy or adapter that stopped is the fastest way to be operating blind. Check the platform health endpoint, which runs background tests on the cluster and lives on the primary VIP on port 30006. And confirm last night’s backup actually wrote to its SFTP target, not that the job was scheduled, that the file landed.
Those checks map to hard thresholds, which is what makes them checkable rather than vibes. The platform health test fails a node when any attached disk breaches 90 percent, so disk headroom is a number you watch, not a feeling. It expects no VM snapshots sitting on the appliance nodes, which is a common and quiet cause of trouble. And it wants the backup connectivity to SFTP working and the scheduled backup written. Put those four numbers where you see them every morning and the daily pass becomes a glance.
The backup that stopped writing
The status grid above is a real week. The backup check went red on day six. The scheduled job still reported that it ran, but nothing was landing on the SFTP target, a credential had rotated on the storage side and the write was failing silently. Because verifying the file actually landed was a daily item, the gap between last good backup and detection was one day. Without that check, we would have found out at the next restore, which is the worst possible moment to learn your backups have been empty. We fixed the credential, confirmed the file wrote, and the cell went green on day seven.
Weekly, monthly, quarterly
The weekly pass keeps the system tidy. I tune whatever alert was noisiest that week so it stops crying wolf, sweep for stale snapshots and idle VMs to reclaim, and glance at configuration drift. Node disk trend belongs here too, because the creep toward 90 percent that the chart shows is a slow-motion problem you want to see at day 20, not day 30. The weekly pass is where you pay down the small debts before they compound.
Monthly is for trends and money. Look at the capacity forecast for any cluster heading toward a cliff, review cost and showback so the numbers stay honest, revisit policies that no longer fit, audit super metrics for duplicates of built-ins, and prune the dashboards nobody opened. Quarterly is for the future: review sizing against object growth, actually test recovery of the analytics cluster rather than assuming it works, and plan the next version move. Each of these draws on an earlier part of this series, which is the point. The runbook is how the individual skills become an operation.
One rule keeps the cadence honest: never move a check to a slower cadence to make a week feel lighter. If something needs a daily eye, a weekly glance will miss it, and the whole value of the cadence is that fast-moving problems get fast attention. The temptation shows up when the daily pass feels repetitive because nothing has gone wrong for a month. That is the pass working, not the pass being unnecessary. The quiet months are exactly what the ten minutes buys, and the moment you demote the daily backup check to weekly is the week the credential rotates and the file stops landing.
| Signal | Healthy value | Act when |
|---|---|---|
| Node disk usage | Well under 90% | Health check fails a node at 90% |
| Snapshots on nodes | 0 | Any snapshot present on an appliance node |
| Last backup to SFTP | Within 24h, file present | No file landed, even if job reported success |
| Collection completeness | 100% | Any site or adapter below 100% |
| Health endpoint (VIP:30006) | Passing | A test fails outside a known day-2 activity |
The quarterly review nobody schedules
Daily and weekly checks catch drift. The quarterly review catches direction. Once a quarter, read the capacity trend for every cluster and note which ones changed slope, revisit the alert definitions that fired most and ask whether each still maps to an action, and run the configuration compliance report end to end instead of glancing at the badge. In 9.1 the continuous compliance option keeps that assessment running rather than waiting for the quarterly pass, but you still want a person to read the trend lines. Write the three things you changed at the bottom of the runbook, so the next quarter starts from a written record.
Wire the checks to signals you already built
A runbook feels like extra work only if every check is a manual hunt. It should not be. Almost every item here maps to something the earlier parts already gave you. Collection completeness and node disk belong on the small operations-of-operations dashboard from the multi-site part. Alert triage is the alerting model, and the noisy-alert tuning is the same symptom work. Reclamation and snapshots are the weekly output of the rightsizing practice. The runbook is not a second system, it is the schedule on which you read the signals you have.
Do the wiring once and the daily pass collapses to a single board. Put completeness, the four daily signals, and a capacity glance on one dashboard scoped to whoever runs the pass. Turn the hard thresholds into alerts so the disk approaching 90 percent pages you rather than waiting for a human to notice on the graph. Where a check can be fully mechanical, like confirming a backup file landed, let an action verify it and raise an alert on failure. What is left for the human is the reading of trends and the judgment calls, which is exactly where a human should be. The mechanical parts run themselves and the person spends their ten minutes on what actually needs a brain.
Make it survive a handover
A runbook proves its worth the week you are on leave. If the daily pass only lives in your head, it stops when you do, and the drift resumes. Write it down as a checklist a competent colleague can run without you, with the signal, the healthy value, and the first move when it goes wrong for each item. Keep it next to the dashboards it references so running it is opening a board, not hunting for one. The best compliment a runbook can get is that someone covered for you and nothing slipped, and you only earn that by making it plain enough to hand over.
If you are planning a version move, the runbook is also your rehearsal script. The checks that confirm health before you upgrade are the same ones you run daily, and the recovery test you do quarterly is the one you want fresh before you touch production. I set out how that folds into an upgrade in the VCF 9.1 upgrade series.
Common questions
How long should the daily pass really take?
About ten minutes once the four signals are on a board you can glance at. If it takes longer, you are hunting for data instead of reading it, so fix the dashboard first.
Where is the platform health check?
The background health tests run on the cluster and the endpoint sits on the primary VIP on port 30006. Some tests can return non-passing during normal day-2 activities, so read a failure in context rather than panicking at the first red.
Why verify the backup file and not just the job?
Because a job can report success while nothing lands, as it did in the example here. Check that a file actually wrote to the SFTP target within the last day. Job status is a promise, the file is proof.
Should I automate the whole runbook?
Automate the mechanical checks and let them alert you. Keep a human on the trends and the judgment calls, because a forecast heading the wrong way is a decision, not a threshold.
What is the one check people skip that bites hardest?
Node disk against the 90 percent threshold. It creeps slowly, it fails the health check when it arrives, and it is trivial to prevent and painful to hit.
How do I keep the runbook from going stale?
Review it every quarter alongside the recovery test. Delete checks that have never caught anything, add checks for whatever surprised you that quarter, and confirm every item still points at a signal that exists. A runbook is a living list, and an unmaintained one slowly stops matching the system it is meant to guard.
Daily pass or shift handover, which comes first?
The daily pass is the anchor. Fold it into whatever handover ritual you already have so it is done once a day at a predictable time by a named person. The worst pattern is everyone assuming someone else looked, which is how a red backup cell survives a weekend unseen.
« Previous: Part 15 | VCF 9 Operations Complete Guide | Next: Part 17 »
References
- Broadcom: VCF Services Platform Cluster Health Checks
- VCF 9.1 Infrastructure Operations
- VCF 9.1 Fleet Management
About The Author
DrJha