By Dr. Pranay Jha, infrastructure architect and long-time vExpert, who designs and tests DR for production VCF estates.
Most disaster recovery runbooks fail their first real test for the same reason. Nobody ran them start to finish before the night the primary site went dark. The document looked complete. It had steps, owners, and a phone tree. Then a power event took out a data hall at 3 a.m., the on-call engineer opened the runbook, and step three said ‘confirm array replication is current’ with no instruction on how to confirm it and no branch for what to do if it was not.
At 3 a.m. the business does not ask for your architecture diagram. It asks two things. When will the service be back, and how much data did we lose. A runbook exists to answer those two questions under pressure, with a tired engineer who did not build the environment, while executives refresh a status page. On VMware Cloud Foundation 9 running VMware Live Recovery, most of the failover is orchestrated for you. The runbook is the human layer wrapped around that orchestration, and that layer is where recoveries go wrong.
What breaks when the runbook meets a real outage
A recovery plan in VMware Live Recovery runs a fixed series of steps for whatever workflow you invoke, whether that is a test, a planned migration, a failover, or a reprotect. You cannot reorder or remove those steps. You can only insert your own message and command steps between them. That constraint is a gift, because it means the mechanics are consistent every time. The danger lives entirely in what you insert and what you assume.
The plan below is a full recovery of roughly 180 virtual machines across five priority groups. Read it as a timeline, not a checklist. The point is that the plan run is only the middle of your recovery time, and the two ends, the decision at the front and the validation at the back, are where teams lose the minutes they never budgeted for.
Structure the recovery plan the way it actually runs
Five priority groups, and what they really mean
VMware Live Recovery gives you five priority groups. All priority 1 machines start before any priority 2 machine, all priority 2 before any priority 3, and so on down to priority 5. Before a group starts, every machine in the group above it must either recover or fail to recover. Inside a group, the product powers on the machines that others depend on first, then brings up as many in parallel as vCenter allows. That parallelism is why tiering matters. Pack too much into priority 1 and you serialize your most important recovery behind machines that could have waited.
Tier for recovery objective, not for neatness. The services that everything else needs, name resolution, authentication, and database primaries, belong in priority 1. Application servers and message brokers that need those databases go in priority 2. Web and mid tier in priority 3. Batch and reporting, which can tolerate a later start, drop to priority 4. Development and low value workloads sit in priority 5, and honestly some of them can stay powered off during the first pass while you protect the RTO on the tiers that earn revenue.
The dependency trap between groups
This is the single most common design error I see reviewed as safe. A tidy tier map with the database in tier 1 and the app in tier 2 looks correct and passes a non-disruptive test, because a test with light load often lets the database finish before the app needs it. Under a real failover with cold caches and heavier boot storms, the timing slips and the assumption breaks. The table below is how I actually assign tiers, with the dependency rule baked into each row.
| Tier | Example workloads | Why here | Dependency note |
|---|---|---|---|
| Priority 1 | DNS, Active Directory, database primaries | everything else needs name resolution and auth | keep a database and its listener in the same group |
| Priority 2 | application servers, message brokers | need the databases answering first | order the broker before the app within the group |
| Priority 3 | web front ends, mid tier services | need the app tier up and healthy | health check the app before this tier starts |
| Priority 4 | batch, reporting, ETL | tolerate a later start | safe to defer, protects RTO on tiers 1 to 3 |
| Priority 5 | dev, test, low value | recover only if capacity allows | consider leaving powered off at first |
Order and timeouts inside a group
Within a single priority group you get finer control, and this is where the real work of a trustworthy plan happens. You can set an explicit startup order so a message broker comes up before the application that binds to it, and you can set a per machine response time, which is how long the plan waits after a machine powers on before it moves to the next one, plus a maximum wait before it gives up and continues. Those timeouts are quiet killers of RTO. Set the response time too long across a large group and you add dead minutes to every boot. Set the maximum wait too short and the plan marches on before a slow database is ready, which recreates the same cold cache problem the tiering was meant to prevent. I tune these against what I see in a timed test, not against a default, because the right value depends on how fast your storage and guests actually boot at the recovery site.
The message step that quietly stalls your recovery
You can insert message steps into a recovery plan. A message step forces the plan to pause until a person acknowledges it in the console. That is useful for a genuine go or no go gate. It is also the most reliable way to freeze your own recovery. Picture a message step that reads ‘confirm storage is presented before continuing’ sitting at 40 percent of the plan while your only engineer is on a bridge call with fifteen people talking. The orchestration is done doing its part. It is waiting on a click that nobody is positioned to make, and your recovery time keeps burning while the plan sits idle.
Reserve message steps for decisions that genuinely must not be automated, and make sure the runbook names one person whose only job during the run is to watch the recovery console and answer prompts. If you find yourself adding a message step so someone can ‘check’ something, ask whether a command step could check it automatically instead. Every pause you add is a place the recovery can stop and wait for a human who is looking somewhere else. For the wider testing discipline behind this, see Part 8 on non-disruptive DR testing.
A worked RTO budget with real numbers
RTO is not the plan runtime. It is decision time plus plan runtime plus validation. Teams quote the 40 minute plan run and forget the 12 minutes spent deciding to declare and the 13 minutes spent proving the services actually work. Budget all three or your stated RTO is a number you have never met. Here is the same recovery from the timeline above, written as a clock against a 120 minute RTO target.
| Phase | Clock | Elapsed | Notes |
|---|---|---|---|
| Declare and decide | 3:00 to 3:12 | 12 min | human call, keep it small and pre-authorized |
| Priority 1 infra (12 VMs) | 3:12 to 3:20 | 8 min | DNS, AD, database primaries |
| Priority 2 app (40 VMs) | 3:20 to 3:30 | 10 min | app servers, message brokers |
| Priority 3 web (60 VMs) | 3:30 to 3:39 | 9 min | web and mid tier |
| Priority 4 batch (50 VMs) | 3:39 to 3:46 | 7 min | batch, reporting |
| Priority 5 low (18 VMs) | 3:46 to 3:52 | 6 min | dev, non-critical |
| Validate and communicate | 3:52 to 4:05 | 13 min | smoke tests, status update |
| Total | 3:00 to 4:05 | 65 min | against a 120 min RTO target |
Run the recovery plan test end to end this quarter, and read the exported history report, not just the green summary. Confirm every priority 1 machine actually answers on the network, because a booted DNS server with the wrong address is still down. Time the test including the decision and validation you would really do, then compare it to your stated RTO. If the test took 90 minutes and your RTO is 60, the RTO is fiction. Confirm the runbook names a specific person for each decision, reachable at 3 a.m., with a named backup. Check that message steps are minimal and that one person owns the recovery console for the whole run.
What I would actually do
Keep the runbook to one page of decisions and let VMware Live Recovery hold the mechanics. I put exactly one message step in the plan, a single go or no go before the power-off, and nothing else. I tier for RTO, which means DNS, authentication, and database primaries in priority 1 and everything that can wait pushed to priority 4 and 5. I disagree with the common advice to protect every workload in one large plan. Split by recovery objective into two or three plans so the tier 1 services never wait behind 150 low value machines. And I rehearse every quarter with the real test workflow, because a runbook nobody has executed is a guess.
For the difference between a planned migration, a failover, and a failback, and when each one is the right call, see Part 9. For how protection groups map into the plans you run here, see Part 6. If you also run day 2 operations on this platform, the VCF 9 Operations series covers the Data Protection dashboard in VCF Operations that reports which machines are protected and recoverable, and the broader VCF 9 series sets the platform context.
Questions I actually get
How many recovery plans should I have, one big one or several?
Split by recovery objective. One plan per tier of service or per application group keeps your tier 1 recovery from waiting on hundreds of low priority machines. Plan your protection groups around how you want to fail over, not just around how the VMs are stored, since a protection group maps into the plans that recover it.
Do virtual machine dependencies work across priority groups?
No. Dependencies are only honored inside the same priority group. Across groups, priority order is all you get, and a lower group waits for the higher group to finish. If two machines depend on each other, put them in one group with an explicit startup order.
What RPO can I actually promise?
With Enhanced vSphere Replication in this release you can set an RPO as low as 1 minute, and it is now the default and only supported site to site vSphere Replication configuration. A 1 minute RPO costs bandwidth and IOPS, so promise it only for the workloads that need it and confirm the schedule holds under real change rates.
Should the runbook include manual steps or trust the automation?
Trust the automation for the mechanics and keep manual steps for decisions only. Every manual step you add is one more thing a tired engineer can get wrong at 3 a.m. Reserve human action for the go or no go call and for validation after the plan finishes.
How do I know my RTO is real?
Time a full recovery plan test, including the decision and validation you would do for real, and compare it to your stated RTO. Read the exported history report for the true per step timings. If you have never timed it end to end, your RTO is an aspiration.
Close the series by rehearsing
This is the last part of the series. If you have read from the start, you have the pieces: replication, protection groups, recovery plans, testing, planned migration and failover and failback, ransomware recovery, cloud targets, and sizing. The runbook is where all of them meet the one morning that counts. Open your recovery plan this week, cut it down to the decisions that need a human, tier it for the objective, and put a real test on the calendar. Then read the timings and fix what the test exposes. The complete VCF 9 disaster recovery guide links every part in order if you need to close a gap before you rehearse.
« Previous: Part 13 | VMware Live Recovery Complete Guide
References
• Recovery Plan Steps, VMware Live Site Recovery 9.0 (Broadcom TechDocs)
• VMware Cloud Foundation Recovery Improvements with VMware Live Recovery (VCF Blog)
• VMware Live Recovery FAQ, January 2026
About The Author
DrJha