, ,

Ransomware Recovery and the Isolated Recovery Environment in VCF 9 (VMware Live Recovery for VCF 9, Part 10)

DR failover restores your last replicated state, which after an attack is often the compromised one. Here is how VMware Live Cyber Recovery uses an isolated clean room and a deep immutable snapshot history to recover VCF 9 workloads from ransomware.

Ransomware Recovery and the IRE
VMware Live Recovery for VCF 9 · Part 10 of 14

By Dr. Pranay Jha, infrastructure architect and long-time vExpert, who designs and tests DR for production VCF estates.

The worst DR runbook for a ransomware event is the one you are proudest of. You press the failover button, your recovery plan powers on the estate at the second site in the order you rehearsed, and forty minutes later the same encryption is spreading through the recovered virtual machines. Replication did its job. It faithfully copied the ransomware to your DR site along with everything else, and your last replicated state is the state the attacker wanted you to keep.

Disaster recovery and cyber recovery are not the same problem, and in VCF 9 they are not the same product. Site to site DR runs on VMware Live Site Recovery, the engine that used to be Site Recovery Manager and vSphere Replication. Ransomware recovery runs on VMware Live Cyber Recovery, the cloud service that used to be VMware Cloud DR. Both live under the VMware Live Recovery umbrella, and the difference between them is the whole point of this part.

The short version. A DR failover restores your last replicated state, which after an attack is often the compromised state. Ransomware recovery instead pulls from a deep, immutable snapshot history, powers candidate snapshots into an isolated clean room disconnected from production, runs behavioral analysis on them, and only then orchestrates the ones that pass back to a protected site. If you own recovery for a VCF 9 estate, this is the workflow you rehearse before you ever need it, and the retention math below is where most plans quietly fail.

Why replication is not a recovery point

Replication has one job: make the copy look exactly like the source. That is precisely what you want for a power failure or a flooded data center, and precisely what you do not want after an intruder has been living in your environment for three weeks. By the time anyone notices the encryption, your continuous replication has already shipped the encrypted blocks to the DR site, and your most recent recovery points are the least trustworthy ones you own.

Ransomware recovery inverts the assumption behind DR. Instead of the newest copy, you want the newest copy that predates the compromise, and you cannot know which one that is until you have inspected several of them. That inspection cannot happen in production, because powering on an infected virtual machine to look at it is how lateral movement spreads. It needs a separate, network restricted place to work. That place is the Isolated Recovery Environment.

Production siteVCF 9 workloadsinfected estateImmutable snapshotsScale-Out CloudFile System (SCFS)Isolated RecoveryEnvironmentrecovery SDDCCarbon Black analysisProtected siteclean workloadsorchestrated backsnapshotspresent read onlyonly if cleanThe clean room path, not the failover path
Immutable snapshots are presented read only to an isolated environment for inspection. Nothing returns to production until it passes analysis. Illustrative schematic.

What the Isolated Recovery Environment actually is

The IRE, which the documentation also calls a clean room, is a recovery SDDC that VMware Live Cyber Recovery stands up on demand with predefined network isolation levels. You do not build it from scratch or stitch together spare hardware and forensic tooling. It is a dedicated, network restricted environment where you can safely power on infected virtual machines, watch how they behave, and decide whether a given recovery point is worth keeping.

Immutable snapshots in the SCFS

The recovery points themselves live off site in the Scale-Out Cloud File System, kept immutable so the attacker cannot reach back and tamper with your history. A protection group can accumulate hundreds of these points. When you launch a ransomware recovery plan, the SCFS presents the snapshots you select directly to the recovery SDDC. They are mounted for inspection, not copied into a running production estate, which is the structural reason the clean room stays clean.

Isolation levels, and the one that surprises people

Each virtual machine powers on inside the IRE attached to a network isolation level. The default is Quarantine plus Analysis, and here is the part that catches teams out: it is not a total air gap. That default deliberately allows a few basic services so analysis can run, including DNS, DHCP and NTP, plus outbound access to the Carbon Black Cloud endpoint that scores the machine. If you assumed clean room meant zero packets in or out, the default will not match your mental model. You can drop a machine to a stricter Isolated level, or build a custom isolation level, but you choose that on purpose.

From the field
The metric that finds the detonation snapshot is entropy rate, not change rate. A busy database or a nightly batch job can push change rate up on any given snapshot, so if you scan the timeline looking for the big change number you will chase false positives all night. Encryption is different: it rewrites files into high entropy ciphertext, and the entropy rate on the snapshot timeline jumps in a way ordinary workload never produces. Read the entropy line first, then use change rate as a supporting clue. The snapshot right before the entropy spike is your best candidate for the last clean point.

Finding the last clean snapshot

VMware Live Cyber Recovery exposes a snapshot timeline with change rate, entropy rate and other metrics for each recovery point. The workflow is iterative by design. You pick a candidate, power it into the IRE under quarantine, let the integrated Carbon Black analysis look for suspicious operating system behavior, malware file signatures and known vulnerabilities, and if it is dirty you try an older snapshot. You repeat until a snapshot comes back clean. The entropy line is what tells you roughly where to start so you are not blindly walking backward one point at a time.

Entropy rate across the snapshot timelineIllustrative values, snapshots every 4 hours0204060%h0h16h28h36detonationlast clean
Entropy holds near baseline, then spikes when encryption starts. The green point just before the spike is the first candidate to validate. Illustrative schematic.
Worked example: does your history reach far enough
Take a protection group with tiered retention: a snapshot every 4 hours kept for 3 days, then one daily snapshot kept for 30 days, then one weekly snapshot kept for 12 weeks. That is 18 plus 30 plus 12, so 60 recovery points, and the oldest reaches back about 84 days. Now suppose forensics later shows the intruder held persistence for 21 days before detonating. Every recovery point newer than 21 days may already carry the implant. The 18 four hourly points are all inside the window and suspect. Of the 30 daily points, 21 are inside the window and 9 predate it. All 12 weekly points predate it. So of 60 recovery points, only 21 are even candidates for clean. A retention plan tuned to your RPO of hours would have kept dozens of points and none of them usable.
Recovery points after a 21 day dwell windowSuspect versus candidate clean, by retention tier0102030184 hourly0 clean219daily12weekly0 suspectsuspect 39clean 21
The same 60 points from the worked example. Depth of history, not point count, decides whether you can recover. Illustrative schematic.

Retention schedule, in numbers

Table columns: Tier, Frequency, Kept for, Points, Candidates after 21 day dwell.
TierFrequencyKept forPointsCandidates after 21 day dwell
ShortEvery 4 hours3 days180
MediumDaily30 days309
LongWeekly12 weeks1212
Total~84 days reach6021

The recovery loop, step by step

Guided recovery is a loop, not a single button. You select a snapshot, validate it in isolation, and either promote it or reach for an older one. The mechanics of the loop matter because the pressure to skip a step is highest exactly when skipping is most dangerous.

SelectsnapshotPower on inquarantineAnalyzeCarbon BlackClean?Orchestrateto productionyesno, pick an older snapshot and repeat
Every candidate is validated in isolation before anything reaches production. Illustrative schematic.
Before you reconnect anything. Orchestrating a recovered virtual machine back to a protected site reintroduces it to your production network. Do it only after the machine has passed analysis in the IRE, keep it isolated until then, and coordinate with your incident response team on timing. A machine that looked clean under quarantine can still hold a dormant payload, so stagger the return and keep monitoring after cutover.

Ransomware recovery versus DR failover

These two workflows share vocabulary, recovery plans and protection groups, which is exactly why teams reach for the wrong one under pressure. The table lays out where they diverge.

Table columns: Dimension, DR failover (Live Site Recovery), Ransomware recovery (Live Cyber Recovery).
DimensionDR failover (Live Site Recovery)Ransomware recovery (Live Cyber Recovery)
Recovery sourceLatest replicated stateChosen point from deep immutable history
GoalRestore newest, minimize data lossRestore newest point that predates the compromise
ValidationBoot and service checksBehavioral and vulnerability analysis in isolation
Network postureRecovered VMs join the recovery networkQuarantine isolation until proven clean
IterationRun the plan onceTry multiple snapshots until one is clean
Right call whenHardware, power or site lossEncryption or intrusion suspected
What to validate before you rely on this
  • Your snapshot retention reaches past a realistic dwell window measured in weeks, not just your RPO window of hours or days.
  • The recovery SDDC region has the capacity to host your protection group when you actually call for it. On demand does not mean unlimited.
  • No private connectivity exists between production and the IRE. No VPN, no Direct Connect, no stretched L2, no shared SDDC group.
  • Carbon Black sensors are deployed where you expect analysis. Without them you lose the integrated behavioral scoring inside the clean room.
  • You have rehearsed a full IRE recovery, not only a DR failover. The two are different muscles.
What I would actually do
Do not wait for a live incident to learn your history is too shallow. Set retention against a dwell window of weeks and keep sparse long term points that reach back months, because a wall of four hourly snapshots is worthless if all of them postdate the intrusion. Rehearse an IRE recovery at least quarterly. The environment is on demand, so a drill costs you a little compute and an afternoon, and it is the only way to find out whether your entropy reading, your Carbon Black sensors and your isolation levels behave the way you assume. For the obvious clean cases in an on premises VCF 9 estate, keep vSAN local immutable snapshots so a fast restore does not need the full cloud round trip. Treat cyber recovery as its own plan with its own retention and its own rehearsal, sitting beside your DR plan rather than hidden inside it.

Questions I actually get

Is ransomware recovery just failing over to my DR site?
No, and treating it that way is how people reinfect themselves. Replication copies whatever is on the source, encryption included, so your latest replicated state is often the compromised one. Live Cyber Recovery instead keeps a deep immutable history and validates candidate points in isolation before you commit.

How far back should my snapshots reach?
Past a realistic attacker dwell window, which is usually weeks rather than hours. If your history only covers your RPO window, every recovery point may already carry the implant. Tier your retention so a few sparse points reach back months.

Does the clean room really have no network at all?
Not by default. The default Quarantine plus Analysis level still allows DNS, DHCP, NTP and outbound access to Carbon Black Cloud so analysis can run. Use the stricter Isolated level or a custom level if you need a tighter posture, and never wire a VPN or Direct Connect from production into the IRE.

Can I do any of this on premises in VCF 9 without the cloud?
Partly. The isolated clean room with integrated analysis runs on a cloud recovery SDDC. On premises you can use vSAN local immutable snapshots for a fast restore of clearly clean points, and Live Site Recovery handles site to site DR, but the forensic isolation and behavioral scoring are the cloud service.

Where you recover to, and why it is not always the cloud

Once a snapshot passes analysis, you have a choice about the destination, and the choice changes your recovery time far more than most people expect. VMware Live Cyber Recovery lets you recover a validated virtual machine directly onto the original protected site, onto a different protected site, or onto the recovery SDDC itself, and later fail back from the recovery SDDC to on premises once the primary is rebuilt and trusted. Each path has a different cost. Landing back on the original site is fastest if that site is clean and rebuilt, but it is also the site that just got hit, so you rarely trust it first. Running on the recovery SDDC keeps you off the compromised estate while forensics finishes, at the price of egress and cloud runtime until you fail back.

The trap is treating the recovery SDDC as a permanent home. It is metered, and a recovered estate left running there for weeks while the security team argues about root cause turns into a bill nobody budgeted. Decide the destination as part of the plan, not in the middle of the incident.

In practice
For the cases where you already know the clean point, an on premises VCF 9 estate has a shortcut worth wiring in ahead of time. VMware Live Cyber Recovery can do a fast restore from vSAN local immutable snapshots, pulling a known good point straight from local storage rather than the full cloud round trip. It is not a substitute for the IRE when you genuinely need to hunt for a clean snapshot, but for a scoped hit where forensics has already named the last trusted point, it turns a multi hour cloud restore into something much shorter. Keep those local snapshots immutable and sized for the workloads you would want back first.

Where this leaves you

The failover button and the clean room solve different failures. Keep DR for the outages that kill hardware, and build a separate cyber recovery plan for the day someone gets inside and stays. Go look at your current snapshot retention today and ask one question: if the intruder had been present for three weeks, how many of these points could you actually trust. If the honest answer is none, that is the gap to close before anything else. Ransomware recovery is a rehearsal you run on a quiet afternoon so that the loud morning is boring.

Next we take the recovery estate off premises entirely and look at DR to the cloud: what changes when your recovery site is a cloud SDDC rather than a second data center you own.

VMware Live Recovery for VCF 9 · Part 10 of 14
« Previous: Part 9  |  VMware Live Recovery Complete Guide  |  Next: Part 11 »

References

About The Author


Discover more from Journal of Intelligent Infrastructure – By Dr Pranay Jha

Subscribe to get the latest posts sent to your email.

Leave a Reply

Your email address will not be published. Required fields are marked *

Architect’s Toolkit

About the Author

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

Discover more from Journal of Intelligent Infrastructure - By Dr Pranay Jha

Subscribe now to keep reading and get access to the full archive.

Continue reading