TL;DR · Key Takeaways
- Firewalls allow and deny; IDS/IPS inspects the allowed traffic for known attack patterns and acts. It is the next layer up from the DFW.
- Distributed IDS/IPS runs in the hypervisor on every host, comparing traffic to signatures. Detect mode (IDS) alerts; Prevent mode (IPS) blocks.
- Malware prevention extracts files from traffic and judges them, known-bad locally and unknown files via network sandboxing.
- ATP combines IDS/IPS, sandboxing, and network traffic analysis, then uses NDR to correlate the noise into a small number of explained attack campaigns.
- This is the vDefend with Advanced Threat Prevention tier (Part 3). Roll it out in detect mode first, tune, then move to prevent. Never start in prevent.
Everything in the security arc so far has been about permission: the firewall decides whether traffic is allowed, and once it is allowed, it flows. But a connection your policy permits can still carry an attack, an exploit aimed at a web server you legitimately exposed, malware riding an allowed file transfer. Firewalls do not look inside the traffic they permit. IDS/IPS and malware prevention do. This is where the NSX security stack moves from allow-and-deny to detect-and-stop, and it is also where teams most often hurt themselves, by switching on blocking before they understand what they are blocking. So the headline lesson is in the rollout, not just the technology.
From firewall to inspection
Think of the security stack as a series of questions asked of the same packet. The firewall asks “is this connection allowed?” If yes, IDS/IPS asks a deeper question: “does this allowed traffic match a known attack pattern?” Malware prevention asks “does this allowed file transfer carry something malicious?” And NDR asks the hardest question of all: “do these individual events, taken together, look like a coordinated attack?” Each layer assumes the one before it passed the traffic and looks for the threats that a simple allow-or-deny decision cannot see. The firewall is necessary and not sufficient; inspection is what catches the attack hiding inside permitted traffic.
Distributed IDS/IPS: detect vs prevent
Distributed IDS/IPS works like the DFW in one important way: it runs in the hypervisor on every host, so inspection is distributed and follows the workload rather than forcing traffic to a central appliance. It compares traffic against signatures, each a pattern describing a known intrusion, and the signature set updates regularly so new threats are covered without you maintaining them. The crucial choice is the mode. In detect mode it is an IDS: it raises an alert when a signature matches but lets the traffic through. In prevent mode it is an IPS: it drops the matching traffic. Same engine, very different operational risk, because a false positive in detect mode is a noisy alert, while a false positive in prevent mode is dropped production traffic.
| Mode | On a match | A false positive means | Use it |
|---|---|---|---|
| Detect (IDS) | Alerts, traffic passes. | A noisy alert to triage. | Always first. Tune here. |
| Prevent (IPS) | Drops the traffic. | A production outage. | Only after tuning in detect. |
You do not enable every signature everywhere. You build profiles that select signatures by severity and attack type, and you scope inspection to the workloads that warrant it using the same groups from Part 13, so a regulated app gets deep inspection while a low-risk internal tool does not pay the cost. NSX 9 also offers performance modes for the inspection engine, so you can trade depth against throughput where it matters.
Malware prevention
Malware prevention extends inspection to files. As files cross the inspected path, NSX extracts them and judges them. A file matching a known-bad verdict is caught immediately. A file NSX has not seen before is sent to a network sandbox, detonated in isolation, and judged by its behaviour, which is how unknown and zero-day malware gets caught when a signature does not yet exist for it. This is meaningfully more than antivirus on the endpoint, because it inspects at the network layer, sees files moving laterally between workloads, and does not depend on an agent being healthy on every VM. It catches the file that the endpoint agent missed or that landed on a system with no agent at all.
ATP and NDR: making sense of the noise
Inspection generates a lot of events, and a flood of individual alerts is its own kind of failure, because no team can triage thousands of them. This is where Advanced Threat Prevention earns its name. ATP combines the detection technologies, IDS/IPS, network sandboxing, and network traffic analysis, and feeds them into Network Detection and Response, which aggregates and correlates the raw events into a small number of explained threat campaigns. Instead of ten thousand alerts, you see a handful of curated attack chains, each showing how the individual events connect into a coordinated intrusion, with the context to understand and respond. NDR is the layer that turns inspection from a noise generator into something a human can actually act on.
| ATP component | What it does |
|---|---|
| Distributed IDS/IPS | Signature-based detection and prevention in the kernel. |
| Malware prevention | File extraction, known verdicts, sandbox for unknowns. |
| Network traffic analysis | Behavioural anomalies the signatures miss. |
| NDR | Correlates events into explained attack campaigns. |
Distributed vs gateway inspection
Just as there are two firewalls, there are two places to inspect. Distributed IDS/IPS runs at the vNIC on every host, so it sees east-west traffic between workloads, the lateral movement an attacker uses once inside, and it is the right tool for inspecting traffic that never leaves the data center. Gateway IDS/IPS runs at the Tier-0 or Tier-1 gateway, like the Gateway Firewall, and inspects the north-south traffic crossing the perimeter. The two mirror the firewall split from Part 14 exactly, and for the same reason: catch the perimeter attacks at the gateway, catch the lateral ones at the vNIC. Most internal threat activity is east-west, which is precisely why distributed inspection matters and why a gateway-only IPS misses the lateral movement that does the real damage in a breach.
A rollout that does not cause an outage
The whole risk of this capability is operational, so the rollout is the design. I run it in four phases, and I do not let anyone skip ahead. Phase one: enable distributed IDS in detect mode on a representative scope, and do nothing else but watch. Phase two: tune, suppress the false positives, adjust the signature profile to the severities you care about, and keep going until the remaining alerts are ones you would actually act on. Phase three: scope precisely, decide which groups genuinely warrant prevention and which only need detection, using the Part 13 groups. Phase four: enable prevent on one scoped, well-understood group, watch it closely, and only then expand. Rushing from phase one to phase four is the single mistake that turns a security win into the incident review.
| Phase | Do | Goal |
|---|---|---|
| 1. Observe | Detect mode on a scope; watch only. | Baseline the alert volume. |
| 2. Tune | Suppress false positives, set profiles. | Alerts you would act on. |
| 3. Scope | Pick groups for prevention. | Right depth per workload. |
| 4. Prevent | Enable IPS on one group; watch. | Block with confidence, expand slowly. |
What I’d Do
Treat this as a program, not a feature toggle. Turn on distributed IDS/IPS in detect mode, scoped to the workloads that justify the inspection, and live with the alerts for long enough to tune the false positives out and trust what remains. Add malware prevention for the file-borne threats your endpoint agents miss, especially on lateral paths. Lean on NDR so your team responds to a handful of correlated campaigns rather than drowning in raw alerts. Only when detect mode is quiet and trustworthy do you move a scoped group to prevent, and even then you watch it. The technology is genuinely strong, but the failures here are operational, not technical: too much, too fast, in prevent mode, is how good threat tooling becomes the thing that caused the outage. Next up is Part 16: the NSX Advanced Load Balancer, Avi, where the platform adds application delivery to networking and security. Are you running IDS/IPS in detect or prevent, and did you earn the right to prevent?
Detection has a real CPU cost, so scope it deliberately
Distributed IDS/IPS is genuinely powerful because it inspects east-west traffic right in the hypervisor, where a perimeter sensor can never see it, but that power is not free. Deep packet inspection against a signature set consumes host CPU, and turning it on everywhere, for every workload, indiscriminately, is how you trade security for a performance problem nobody scoped. The right instinct is the opposite of blanket coverage: apply IDS/IPS to the workloads and the flows where it actually buys you risk reduction, the crown-jewel applications and the sensitive zones, and leave the rest uninspected until there is a reason to change that.
Scoping the inspection is the same applied-to discipline that governs the firewall. You decide which segments and which groups the IDS/IPS profiles attach to, so the cost lands where the value is rather than spread thin across everything. This is also where the licensing reality from earlier in the series shows up, because these advanced security capabilities sit under the vDefend entitlement, so the scope of your inspection is a budget decision as much as a performance one. Inspect what matters, account for the host CPU it consumes, and you get the lateral threat detection without the platform-wide tax that comes from switching it on everywhere by default.
Detect first, then prevent
The safest way to operationalize IDS/IPS is to separate detection from enforcement in time. Start in detect mode, where the system tells you what it would have blocked without actually blocking anything, and live there long enough to learn your real traffic. Every environment generates false positives, benign patterns that look suspicious to a generic signature, and discovering those in detect mode costs you nothing but a log entry. Discovering them in prevent mode costs you a blocked application and an angry user, so the order matters.
Once you have tuned away the false positives for a given set of workloads, you move those segments to prevent mode, where the system actively stops what it detects. Doing this segment by segment, rather than flipping the whole estate at once, keeps the blast radius of any tuning mistake small and gives you a clean rollback. The pattern mirrors the rest of the security methodology in this series: earn the visibility first, validate against reality, then tighten enforcement deliberately. Detect, tune, prevent, one zone at a time, and intrusion prevention becomes a control you trust rather than a source of mysterious outages.
Signatures have a lifecycle too
Distributed IDS/IPS is only as good as the signatures behind it, and signatures are not a set-and-forget asset. New threats appear, signature sets are updated to catch them, and a sensor running stale signatures is quietly blind to exactly the attacks that emerged after you deployed it. Keeping the signature set current is part of operating the capability, not an optional extra, because the value proposition of intrusion detection collapses if the detection content ages out.
Currency has to be balanced against stability, especially in prevent mode. A signature update can change what gets flagged or blocked, so the same detect-then-prevent caution that governs the initial rollout applies to updates: validate that a new signature set does not start dropping legitimate traffic before you let it enforce across sensitive segments. The mature pattern is to keep signatures current on a regular cadence, review what each update changes, and roll it through detect-then-prevent the way you rolled out the capability in the first place. Fresh signatures catch new threats; validated signatures avoid new outages; you want both.
References
- Distributed IDS/IPS and Malware Prevention for vDefend (Broadcom TechDocs, 9.0)
- VMware vDefend Advanced Threat Prevention
- NSX 9 Distributed Firewall Fundamentals (NSX Series, Part 12)
About The Author
