Build Protection Groups and Recovery Plans in VCF 9 That Hold Up at Failover (VMware Live Recovery for VCF 9, Part 6)
Protection groups decide what fails over together and recovery plans decide how it comes back. A field guide to membership, five priority levels, dependencies and timed testing in VCF 9.
By Dr. Pranay Jha, infrastructure architect and long-time vExpert, who designs and tests DR for production VCF estates.
The replication was green for months. Then the primary site lost power, someone clicked failover, and forty machines came up in a jumbled order. The database tried to start before its storage was promoted, the app tier came up pointing at an IP range that no longer existed, and the one virtual machine that ran the license server was not in any recovery plan at all, so it never came up. Replication had done its job perfectly. The recovery had no plan behind it.
Replication copies bytes. It does not decide what fails over together, in what order, onto which network, or whether anyone is watching. That job belongs to two objects you build once and lean on at the worst possible moment: the protection group and the recovery plan. This part is for the admin or architect who will sign that plan and then have to run it while the business asks how much longer.
Bottom line: a protection group answers what travels together, a recovery plan answers how it comes back, and almost every ugly failover I have seen came from one of two gaps, a replicated machine that was in no protection group, or a recovery plan whose ordering was never timed against a real test.
Naming note for 2026. The engine here is still Site Recovery Manager plus vSphere Replication, now sold as VMware Live Recovery. In VCF 9.1 the same capability is folding into the platform under the name VCF Protection and Recovery, and the standalone Site Recovery Manager and vSphere Replication appliances are deprecated in 9.1. The concepts of protection group and recovery plan carry across the rename unchanged, so everything below still applies. Confirm the exact build you run before you quote a version.
A protection group is a decision about what travels together
A protection group is the unit of failover. Every virtual machine you want to recover has to belong to one, and the group is what a recovery plan actually points at. If a machine is replicating but sits outside every protection group, it will not appear in any plan, and it will not come back when you run one. That sounds obvious written down. It is the single most common thing missing at 3 a.m., because machines get provisioned long after the DR design was signed.
Three ways to decide membership
How a machine joins a group depends on the replication type. For host-based Enhanced vSphere Replication you build a virtual machine protection group and add machines to it. For array-based replication you build a datastore group, and membership follows the LUN, so every machine on that datastore comes along whether you meant it to or not. VCF 9.1 also lets you assign machines by vSphere tag, with match-any or match-all logic, so a workload is protected the moment it is provisioned with the right tag rather than the next time someone remembers to edit the group.
Membership is the gate. A replicated machine with no protection group never enters a recovery plan, so it never recovers.
In practice. Tag-based membership is the fix for the license-server problem, but it moves the risk rather than removing it. Now the question is whether the tag was applied. Bake the tag into your VM provisioning template or a policy, not into a person remembering to click it, and then audit the tag on a schedule.
A recovery plan is a runbook the machine executes for you
A recovery plan runs a protection group as an ordered sequence. It handles the pre-work you would otherwise do by hand under pressure: it promotes the replicated storage, applies your network mappings so recovered machines land on the right port groups, applies re-IP rules where the recovery site uses a different subnet, and then powers machines on in the order you set. It can pause on a prompt step so a human confirms something, and it can call scripts. Done well, the plan is boring, and boring is the goal when the site is down.
Priority levels, not a free-for-all
Recovery ordering is built on five priority levels, where priority 1 starts first and priority 5 starts last. Every machine in a new plan defaults to priority 3, which means if you change nothing, everything comes up at once in the middle band. The plan will not start a lower priority level until the higher one has finished, and finished has a specific meaning here: the tool waits for a VMware Tools heartbeat from each machine before it counts that machine as up. So your database tier does not just power on before the app tier, it is running its guest OS before the app tier is even asked to boot.
Dependencies live inside a priority level, not across them
Inside a single priority level you can also set explicit dependencies and a per-machine start order. Here is the trap. A dependency is only honored when both machines share the same priority level. If you set a dependency from a priority 3 machine onto a priority 1 machine, it is silently ignored, because the priority ordering already covers that case. People discover this when a machine that was supposed to wait for another does not wait, and the log gives no error. Use priority levels for the coarse tiering between application layers, and reserve dependencies for fine ordering within one tier.
From the field: the dependency that never fired
A three tier app kept coming up with the middleware dead after every test. The dependency from middleware onto the database was configured and looked correct in the UI. The database sat at priority 1 and middleware at priority 3, so the tool treated the dependency as redundant and ignored it, then started middleware the instant priority 3 began, while the database was still finishing recovery inside priority 1 and had not yet accepted connections. The fix was not a bigger timeout. It was moving both machines into the same priority level and letting the dependency do the ordering, or trusting the priority gap and adding a start delay. Dependencies and priority levels solve the same problem at different scales, and mixing them is how you get ordering that reads correct and behaves wrong.
What the plan actually costs in minutes
Recovery time is the sum of serialized phases, each waiting on the one before it. Because each priority level waits for the one above it to report healthy, the plan is only as fast as its slowest tier plus the fixed pre-work. Here is a worked example for a fifty machine estate split across three tiers. The numbers come from timing a test, not from a datasheet, and yours will differ, but the shape holds.
Serialized phases. Each tier starts only after the one above reports healthy, so the plan total is additive, not parallel.
The lesson in the table is that adding machines to the web tier barely moves the total, because they boot in parallel inside priority 5. What moves the total is adding another serialized tier, or letting one slow machine in priority 1 hold the whole plan open while everything below it waits.
Minutes per phase from the worked example. The app tier, not the largest tier, is the longest pole.
What to validate before you rely on this
Run a test failover and time each tier with a stopwatch, then compare it to the RTO you promised the business. Confirm every protected machine appears in exactly one recovery plan by reconciling the protection group membership against your CMDB or VM inventory. Check that network mappings and re-IP rules point at port groups and subnets that still exist on the recovery site. Verify that any prompt step or script step will not silently pause an unattended failover forever. Last, confirm dynamic tag membership actually captured a machine you provisioned this week, not just the ones present at design time.
Before you test, it helps to see the design choices in one place. This is how I answer the recurring protection group and recovery plan questions, and where I land by default.
Table columns: Design question, Common option, What I choose, Why.
Design question
Common option
What I choose
Why
What defines a protection group?
the datastore it sits on
the application it serves
recovery is per service, not per LUN
Groups per recovery plan?
many groups in one plan
one plan per service tier
tier 1 never waits on bulk workloads
How to assign membership?
a manual VM list
tags, then audited
a manual list is where forgotten VMs hide
Reprotect scope after failover?
the whole site at once
per group
a smaller blast radius if one group stalls
Related in this series: the replication choice underneath these groups is covered in Part 5, and running the plan for real is Part 9.
Testing the plan without touching production
The reason you can rehearse this at all is that a test failover runs in an isolated bubble. The plan recovers your machines from the latest replica onto an isolated test network defined by the plan test mapping, while the protected machines keep running and replication keeps flowing. You get to watch the real recovery boot order, log into the recovered database, and confirm the app comes up, all without cutting over the business. When you are done you run Cleanup, which powers off the test machines and restores the placeholders, leaving no residue.
Test the isolation, not just the boot. The classic mistake is mapping the test network to something that can still reach production, so a recovered domain controller starts talking to the live one and you spend the afternoon cleaning up replication conflicts. The bubble only protects you if it is actually sealed.
A test failover recovers into a sealed network. If that seal leaks toward production, the rehearsal becomes the incident.
Before you run a real failover. A real recovery plan, unlike a test, powers down or promotes storage and cuts the workload over for real. It changes production. Run the test failover and Cleanup path until it is boring first, and confirm your reprotect and failback story before you commit, because a plan you can only run in one direction is half a plan. Never rehearse for the first time during the outage.
The setup I would ship
I do not build one giant recovery plan for the whole estate, even though the product happily lets you. I build several smaller plans aligned to application service boundaries, so I can fail over the payments stack without dragging the analytics cluster with it, and so a test of one service does not require a maintenance window for everything. A monolith plan looks tidy in the console and turns every partial recovery into an all-or-nothing gamble.
For ordering, I lean on the five priority levels and keep dependencies rare. Priority tiering with the Tools heartbeat gate handles the real requirement, which is that the database is answering before the app tier boots. Fine grained dependency chains inside a tier are brittle, they read well in the design document and fail quietly in the run. And I map protection group membership to tags wherever the platform allows it, then audit those tags, because the failure mode I fear most is a machine that was never added to the plan at all, which no amount of plan tuning will ever catch.
Questions I actually get
Can one virtual machine be in more than one protection group? No. A machine belongs to a single protection group, which is why membership design matters so much. If two applications share a machine, you have to decide which recovery story owns it, and that is usually a signal to separate the workloads.
Can one recovery plan cover several protection groups? Yes, and that is the normal case for array-based setups where each datastore is its own group. A plan can pull in multiple groups and order all their machines together with priority levels and dependencies across the combined set.
Why did my dependency get ignored? Almost always because the two machines were in different priority levels. Dependencies only apply within a single priority level. Put both machines in the same level, or rely on the priority gap and remove the dependency.
What happens to a machine that is replicating but in no protection group? Nothing, during a recovery. It is not in any plan, so it does not fail over. This is the most common and most damaging gap, and it is exactly what dynamic tag membership is meant to close.
Does a test failover affect my RPO or replication? No. The test recovers from the current replica onto an isolated network while replication continues normally. Cleanup then removes the test machines. Your protected workload and its replication are untouched throughout.
Build the membership so nothing important is orphaned, build the ordering so tiers wake in sequence, and then prove both by timing a real test. A recovery plan you have never timed is a hope, not a plan. Next in this series we move from single-site mechanics to networking for DR, where re-IP, mappings and the test bubble decide whether recovered machines can actually talk to each other.
Your move: open your recovery plan, pick the one tier you have never timed, and run a test failover against it this week. If you want the wider picture, the VMware Live Recovery for VCF 9 guide maps where this fits, and Part 5 on array-based vs host-based replication explains why your protection group type is decided before you ever open the plan.
Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.
Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.
DrJha