By Dr. Pranay Jha, infrastructure architect and long-time vExpert, who designs and tests DR for production VCF estates.
A recovery plan that has passed its scheduled test every quarter for two years can still fail the first time it runs for real. That is not a paradox. A test recovery and a real failover diverge at the two points that matter most, and a clean test result tells you almost nothing about either one. The test is still worth running. It just proves less than the green checkmark suggests, and the job of this part is to show you where the proof stops and what to do about the gap.
If you own DR for a VMware Cloud Foundation 9 estate, non-disruptive testing is the one recovery activity you can run any week without a maintenance window, without paging anyone, and without touching production. That is its appeal and also its trap. Because it never touches production, it never exercises the parts of recovery that only exist in production. This part is for the admin who has to sign off on a plan and then stand behind it at 3 a.m., not just admire a wall of green.
THE SHORT VERSION
A test recovery powers your protected VMs up on an isolated network, running off a temporary snapshot or redo log at the recovery site, while production keeps running and replication keeps flowing.
It exercises boot order, dependencies and custom scripts. It does not power down the protected site, it does not promote storage for real, and it runs on a fenced network, so re-IP, DNS and routing stay unproven.
Test often, run cleanup every single time, and never leave a test open over a long weekend.
How a test recovery actually runs
The design goal of a test is simple to state and easy to underestimate: bring the protected workloads up at the recovery site, in the right order, without the protected site ever noticing. VMware Live Site Recovery, the engine sold inside VMware Live Recovery and folded into VCF 9.1 as the platform recovery capability, does this by never committing anything. It builds a throwaway copy of the recovered state, runs your machines against it on a network that goes nowhere, then throws the copy away. Confirm your exact build before you quote any of this to an auditor, because the product names moved recently even though the behavior did not.
The throwaway copy is the key. If you replicate with vSphere Replication, the test creates redo logs on the replica disks at the recovery site. Your test VMs write into those redo logs while the base replica keeps receiving live changes from production underneath. If you replicate with an array, the array snapshots the volumes holding the replica disks and the test runs off the snapshot while replication continues into the live volume. Either way, replication is not paused and the recovery point keeps advancing during the test.
What the plan does, step by step
A test run walks almost the same sequence as a real recovery, minus the two irreversible steps. In order:
1. It checks preconditions. Replication must be healthy and any previous test must have been cleaned up. If the last test was never cleaned, you cannot start a new one.
2. You choose whether to synchronize recent changes first. Syncing gives the test the freshest possible data at the cost of a few minutes up front. Skipping it uses the last replicated state.
3. It stands up the temporary copy. Redo logs for vSphere Replication, or an array snapshot for array-based replication, so the base replica keeps receiving production changes throughout.
4. It powers on the recovered VMs into the test network in priority order, honoring dependencies and running any pre-power-on and post-power-on scripts. Recovery-site machines that the plan is configured to suspend get suspended, exactly as they would in a real run.
5. It pauses and waits. The plan stops at the end and holds the environment up so you can log in and validate. Nothing cleans up until you say so.
6. You run cleanup. VMs power off, get swapped back to placeholders that preserve their identity, and the redo logs or snapshots are discarded. Replication continues without interruption.
The timeline below is a real-shaped run for a forty VM plan across three priority tiers. The exact minutes vary with your storage and boot dependencies, but the shape holds: power-on is quick, validation is where the wall-clock time goes, and cleanup is not free.
What a passing test proves, and the two things it never touches
A test exercises nearly every step of the plan. Boot order, dependency waits, VMware Tools heartbeats, custom scripts, priority tiers, resource and folder placement, and whether the machines even power on from the replica: all real, all proven. That is genuinely useful, and it is why a test that fails is worth its weight in gold. It found something before the outage did.
The trouble is the two steps a test deliberately skips, because they are the two that cannot be undone. It never powers down the protected VMs, and it never forces the recovery site to take real ownership of the replicated storage. On top of that, it runs on an isolated network instead of production. Those three concessions are exactly the mechanisms that break during real events. Storage promotion can stall or surface a device it cannot mount. The production network forces re-IP, DNS updates and routing changes that the isolated test network never asks for. A test cannot see any of it.
| Step in the plan | Test recovery | Real failover |
|---|---|---|
| Protected VMs powered down | No, they keep running | Yes, in a planned migration |
| Storage ownership | Temporary snapshot or redo log | Recovery site promotes the real volume |
| Network the VMs land on | Isolated test network | Production network, with re-IP and DNS |
| Replication during the operation | Continues normally | Stops once the site is recovered |
| Reversible | Yes, run cleanup and it is gone | No, you commit to a failback later |
FIELD NOTE
A team I worked with ran a clean quarterly test for over a year and treated it as proof the plan worked. During an actual site loss, every VM powered on and the console went green in about twelve minutes. The service stayed down for another ninety while people rebuilt DNS by hand, because the recovery site used different subnets and the test network had never made anyone confront that.
The test was not wrong. It was answering a narrower question than anyone thought it was. Networking was the layer that bit, which is why Part 7 on DR networking pairs with this one so tightly.
The cost of leaving a test running
Because a test is non-disruptive, it feels free, so people leave them open. Someone starts a test on a Friday to validate a change, gets pulled away, and the test sits there until Monday. It is not free. While the test runs, production keeps changing and those changes keep replicating into the base replica underneath your redo log or behind your array snapshot. The temporary copy grows for every hour the test stays open, and it grows at the rate your protected workloads change.
Working the numbers
Take a protected workload of 4 TB with a daily change rate of 3 percent. That is 120 GB of change every day flowing into the recovery site while the test holds its temporary copy open. If the recovery datastore has 800 GB of free space, the arithmetic is unkind: the copy consumes it in a little under seven days, and the copy fills the datastore and replication itself is what breaks next, because it now has nowhere to land.
WORKED EXAMPLE
Protected data under the plan: 4 TB. Daily change rate: 3 percent. Daily growth of the temporary copy: 4000 GB times 0.03 equals 120 GB per day.
Free space on the recovery datastore: 800 GB. Days until it fills: 800 divided by 120 equals about 6.7 days.
Leave a test open across a long weekend plus a holiday Monday and you are inside that window. The fix is not more storage. It is a habit: validate, then clean up.
| Days test left open | Copy consumed (GB) | Share of 800 GB free |
|---|---|---|
| 1 | 120 | 15 percent |
| 3 | 360 | 45 percent |
| 5 | 600 | 75 percent |
| 6 | 720 | 90 percent |
| 7 | 840 | full, replication stalls |
BEFORE YOU TOUCH CLEANUP
Cleanup discards the test copy and everything written inside it. That is the point, but it means anything you wrote into a test VM to prove a repair is thrown away too. Capture your evidence, screenshots, logs, query results, before you run it.
Cleanup can also fail with a datastore that will not unmount if a host already dropped it. Resolve that before starting the next test, because you cannot run another test or a real failover until cleanup completes successfully.
How often to test, and why annual is theater
Here is where I part ways with the common advice. Plenty of DR guidance, and plenty of audit checklists, treat an annual DR test as a passing grade. I think an annual test proves almost nothing, because the thing it validated is gone long before you test again. VMs get added and removed, dependencies shift, someone re-IPs a subnet, a new tier arrives without a protection group. A plan is only as current as your last test against the real inventory, and a year of drift is a year of untested change.
My position is monthly at a minimum for anything that matters, and after any significant change to the protected estate. The test is non-disruptive by design, so the only real cost is the hour of attention it takes to run it and clean it up. The chart makes the drift concrete: the taller the bar, the longer a broken plan can sit undetected.
MY CALL
Test monthly for tier-one plans and after every significant change to the protected inventory. Automate the run and the cleanup so the only human step is validation.
Once a year, go further than the isolated test allows. Schedule a controlled window and fail a small, non-critical protection group over onto the real recovery network, so re-IP, DNS and storage promotion get exercised for real at least once a year. The isolated test tells you the plan runs. Only a real-network run tells you the service comes back.
WHAT TO VALIDATE BEFORE YOU RELY ON THIS
That the test network is genuinely isolated. A misconfigured test network that touches production can duplicate live IP addresses and cause the outage you were trying to avoid.
That cleanup completed after the last run. Check the plan state, not your memory. A plan stuck in a test state blocks the next test and a real failover alike.
That the test data is fresh enough. If you skip the pre-test sync, know how stale the last replicated point is before you draw conclusions from it.
That your recovery datastore has headroom for the copy to grow. Size it for the longest test you might realistically leave open, not the ideal one you plan.
Questions I actually get
Does running a test hurt my RPO or interrupt replication?
No. Replication keeps flowing throughout the test, into the base replica underneath a redo log or behind an array snapshot, and your recovery point keeps advancing. The one caveat is storage. Leave a test open long enough and the growing temporary copy can fill the recovery datastore, and that is what stalls replication, not the test itself.
My test passes every time and I am still nervous. Am I being paranoid?
You are being appropriately careful. A test runs on an isolated network against a temporary copy, so it never exercises re-IP, DNS, routing, or real storage promotion. Those are the steps that fail during actual events. A perfect test record is a narrower claim than it looks, and it guarantees less than most teams assume.
Do I really have to run cleanup, and what happens if I forget?
Yes, cleanup is mandatory. You cannot start another test or run a real failover until the previous test has been cleaned up successfully. Beyond blocking you, an uncleaned test keeps its redo logs or snapshots growing, so a forgotten test quietly eats recovery-site storage until someone notices.
Can I test with zero impact to production?
That is the whole design. The protected site keeps running and replication continues. The only machines affected are recovery-site VMs that your plan is explicitly configured to suspend during a recovery, and they are suspended for both tests and real runs.
Can I trust a test to validate my application, not just the VMs?
Partly. A test proves the machines boot in order and your scripts run, and you can log into the test network and check application health there. What it cannot prove is that clients on the production network can reach the app after a real failover, because that path does not exist inside the test bubble.
Where this leaves you
Non-disruptive testing is the cheapest confidence you can buy in DR, and the easiest to overvalue. Run it often, because drift is the real enemy and a monthly cadence keeps the blind window short. Clean up every time, because a forgotten test is a slow storage leak with your recovery point attached. And at least once a year, break out of the isolated network and prove the service actually comes back, not just that the machines power on. A green test is a good day. It is not the same as a recovered business.
Next in the series we move from testing to doing: planned migration versus failover versus failback, and how the choice changes what you can promise. If you want the map of the whole series, the VMware Live Recovery for VCF 9 guide collects every part. For how DR fits the wider platform, see the VCF 9 Operations series and the VMware Cloud Foundation 9 guide. Then go run a test this week, and this time watch what it does not tell you.
« Previous: Part 7 | VMware Live Recovery Complete Guide | Next: Part 9 »
References
Testing a Recovery Plan, VMware Live Site Recovery 9.0, Broadcom TechDocs
Clean up After Testing a Recovery Plan, Broadcom TechDocs
VMware Live Recovery Solution Overview
About The Author
DrJha