By Dr. Pranay Jha, infrastructure architect and long-time vExpert, who designs and tests DR for production VCF estates.
Compute is the easy part of a failover, and this one looked flawless. Storage promoted, the recovery plan stepped through its priority tiers, every virtual machine reported powered on, and the console was a wall of green. Then the phones started. The order-entry app was up but unreachable, because it had come back on an IP range that only existed at the site now sitting dark. Another two tiers were reachable, yet every client still resolved their names to addresses on the failed site, so nothing routed. The compute recovered in about eleven minutes. The service stayed down for another hour and a half while people chased DNS records and static routes by hand.
Replication moves the bytes and the recovery plan powers the machines. Networking decides whether anyone can use what you recovered, and it is the layer of DR that fails most quietly. A test recovery runs on an isolated network and passes every single time, which is precisely why the mapping and re-IP gaps stay invisible right up until the day you fail over onto the real production network. This part is for the admin who signs the plan and then has to explain, at 3 a.m., why the app is green but nobody can log in.
WHO THIS IS FOR
You have replication running and a recovery plan built, but you have never pinned down which network the recovered machines actually land on.
Your recovery site uses different IP subnets than production, and you are still undecided between re-IP and stretching the network.
Your DR tests always pass, and some part of you suspects that is the problem.
One note on names before the mechanics. The engine here is Site Recovery Manager plus vSphere Replication, sold as VMware Live Recovery, and in VCF 9.1 the same capability folds into the platform as VCF Protection and Recovery. Network mappings, IP customization rules and test networks carry across the rename with the same behavior, so what follows holds whichever build you run. Confirm your exact version before you quote a number to anyone.
The mappings a recovery plan quietly depends on
Before a recovery plan can bring a machine up somewhere useful, it has to know four things about the destination: which network the vNIC attaches to, which folder the machine lands in, which compute resource runs it, and which throwaway network to use during a test. These are the inventory mappings. Get them right once and the plan reuses them for every run. Miss one and the plan will still run, it will just put the machine in the wrong place, and you will find out at the worst time.
Network mapping
A network mapping ties a protected-site port group to a recovery-site port group. When the plan recovers a machine, its vNIC gets reconnected to whatever the mapping points at. If the protected VM sat on a segment called Prod-App and the mapping sends it to a recovery segment on a different VLAN and subnet, the machine comes up connected but with an address that no longer fits. That mismatch is the root of almost every re-IP decision you will make later. The mapping decides where the cable plugs in; it does not fix the address on the far end.
Folder and resource mappings
Folder and resource mappings are less dramatic but bite in their own way. The resource mapping decides which cluster or resource pool runs the recovered machine, and if that target is undersized, your recovery admission-control math falls apart the moment a real failover lands a few hundred machines on it at once. The folder mapping matters because your permissions, tags and automation often key off folder placement. A machine that recovers into the wrong folder can come up outside the policies that were supposed to protect it.
The test network
The test network is the one mapping that only ever gets used during a test, which is exactly why it is easy to set carelessly. During a test recovery, machines attach to this network instead of the real recovery network. Live Recovery can auto-create an isolated network for you, or you can point the test at a real port group. Point it at a real port group that reaches production and you have built a loaded gun, because a test failover will bring up duplicate copies of production machines with production addresses on a network that can actually route.
| Mapping | What it decides | What breaks if it is wrong |
|---|---|---|
| Network | Which port group the vNIC attaches to | Machine up, address unreachable |
| Resource | Which cluster or pool runs it | Contention or admission failure at scale |
| Folder | Where it lands in inventory | Falls outside tags, permissions, policy |
| Test network | Where test machines attach | Test leaks onto production if not isolated |
Re-IP, the step that adds minutes you did not budget
If your recovery site cannot present the same subnets as production, recovered machines need new addresses. Live Recovery can do this for you through IP customization rules, and the mechanism is worth understanding because it decides how much time re-IP steals from your RTO.
How IP customization rewrites an address
An IP customization rule works on subnets, not individual machines. You define a rule per network mapping that says, for a statically addressed machine on the protected subnet, keep the host portion of the address and swap the network portion for the recovery subnet. A machine at 172.20.10.155/24 becomes 172.20.110.155/24. The host bits, that trailing 155, are preserved; the network bits are replaced. One rule covers every static machine on that mapped network, so you are not maintaining a spreadsheet of per-adapter addresses. Turn on the advanced setting that applies the mapper automatically and the plan evaluates these rules during recovery without you touching each machine.
What re-IP will not do for you
The customization is only as good as the guest it runs inside. It needs VMware Tools running and a supported guest OS, because the address change happens through guest customization after power-on. DHCP machines are ignored, since they will pull a fresh lease anyway. Machines with several NICs, hard-coded addresses buried in application configs, or Linux guests with network settings the customization engine does not manage will not come out the far side correctly. And every static machine that does get re-IP-ed needs a guest reboot to apply the change, which is the part that costs you time.
FROM THE FIELD
The re-IP gotcha that surfaces only in a real event is the hard-coded address that lives outside the guest network stack. IP customization fixes the interface, so the machine pings, and the plan turns green. Then the app fails to start because a database connection string, a cluster witness entry or a monitoring agent config still points at the old production IP. The network layer recovered, the application config did not, and no test on an isolated bubble ever exercised those cross-machine references. Re-IP is as much an application-inventory task as a networking one, and that second half is where people get surprised.
The test bubble, and why a passing test can still lie
The isolated test network is the best feature of orchestrated DR and the one that lulls people into false confidence. You can run a recovery plan in test mode as often as you like, machines boot on a fenced network with no uplink, and production keeps running untouched. That is genuinely useful. The trap is treating a green test as proof the real failover will work, when the test deliberately skips the two things most likely to break: the real recovery network and re-IP against live DNS and routing.
Where the test and the real run diverge
In a test, the plan applies test settings, drops machines into the bubble, and never advertises a route to the rest of the network. In a real failover it applies the recovery mapping, re-IPs onto the production recovery subnet, and reboots each static machine to take the new address. That reboot loop runs in batches and adds real minutes, and none of it is exercised by the isolated test. If you want a test that means something, occasionally run it against the actual recovery network mapping during a maintenance window, with the source side gracefully shut down first, so re-IP and DNS behavior get exercised for real.
WORKED EXAMPLE
Take a plan of 180 statically addressed machines that all need re-IP because the recovery site is on new subnets. IP customization reboots each one to apply the address. If the plan cycles them in batches of 20 and each re-IP-and-reboot cycle averages 90 seconds, that is nine batches at 90 seconds, so 810 seconds, roughly 13 and a half minutes added on top of storage promotion and power-on ordering.
That 13.5 minutes is invisible in every isolated test, because tests do not re-IP against the production subnet. It only appears when you fail over for real, which is why an RTO signed off from test timings alone tends to be optimistic by exactly this margin.
Before you run anything against the real recovery network. A test that uses the production recovery mapping instead of the isolated bubble can bring machines up with live addresses. If the source side is still running, you risk duplicate IPs and split-brain on shared services. Only do this with the protected machines cleanly powered off, a maintenance window agreed, and someone watching DNS. When in doubt, keep the test in the isolated bubble.
Stretch the layer 2 or re-IP? The call I make
Two roads lead out of the subnet-mismatch problem. Stretch the network so the same subnets exist at both sites and machines keep their addresses, or accept different subnets and re-IP on recovery. The table lays out what each costs.
| Approach | Re-IP needed | RTO added | Main dependency |
|---|---|---|---|
| Stretched layer 2 | No | 0 min | Stretched fabric, risk of shared failure domain |
| Routed sites, IP customization | Yes, automated | ~13.5 min at 180 VMs | VMware Tools, DNS TTL, guest reboot |
| Routed sites, manual re-IP | Yes, by hand | ~90 min at 180 VMs | People and time you do not have at 3 a.m. |
WHAT I WOULD ACTUALLY DO
Here is where I part ways with the common advice to stretch layer 2 everywhere so DR is address-transparent. A stretched fabric across sites is a shared failure domain, and I have watched a broadcast storm or a control-plane issue take out both sites at once, which is the exact thing DR exists to survive. For most environments I would route the sites, keep them independent, and lean on IP customization with a tested set of rules. Reserve stretched layer 2 for the handful of workloads that truly cannot re-IP, such as clustered databases with address-pinned quorum, and put everything else on re-IP. Independence at the network layer is worth a few minutes of reboot time.
WHAT TO VALIDATE BEFORE YOU RELY ON THIS
Confirm every protected machine that needs a new address has VMware Tools running and a guest OS the customization engine supports; the ones that do not are your manual list, so build it now, not during the event.
Check your DNS TTLs on the names that must move. A 24-hour TTL means clients keep resolving to the dead site long after failover, so lower the TTLs on DR-critical records ahead of time.
Verify the test network is genuinely isolated. If it can reach production, one test run can collide with live addresses, so prove the bubble has no uplink before you trust it.
Myth versus reality
| What people assume | What actually happens |
|---|---|
| A passing test means the failover will work. | Tests run in an isolated bubble and skip re-IP against live DNS and routing, the two things most likely to break for real. |
| IP customization fixes the whole machine. | It fixes the interface. Hard-coded addresses in app configs and connection strings still point at the old site. |
| Re-IP is instant. | Each static machine reboots to take its new address, and across a large plan that adds real minutes to RTO. |
| Stretching layer 2 is the safe default. | A stretched fabric is a shared failure domain that can take both sites down together. It is a targeted tool, not a blanket answer. |
Networking is the DR layer nobody watches until it fails, because replication has dashboards and recovery plans have green checkmarks while mappings and re-IP just sit there quietly until you fail over for real. Map the four inventory targets deliberately, decide re-IP versus stretched layer 2 with the RTO cost in front of you, prove your test bubble is truly isolated, and run at least one test a year against the real recovery mapping so the numbers you signed are the numbers you get. Do that and the app comes up green and reachable, which is the only kind of green that counts.
For the objects these mappings attach to, see the previous part on protection groups and recovery plans. For the wider platform picture, the VCF 9 Operations series and the VCF Automation guide cover the day-2 and provisioning sides. The full DR path lives in the VMware Live Recovery for VCF 9 complete guide.
« Previous: Part 6 | VMware Live Recovery Complete Guide | Next: Part 8 »
References
- VMware Live Recovery: Customizing IP Properties for Virtual Machines
- VCF Protection and Recovery 9.1 Release Notes
- VCF 9.1: Integrated Cyber Recovery and Network Isolation
About The Author
DrJha