During a recent VCF 9 deployment workshop, I told the customer they needed specific MTU settings. He was not from a networking background, so he asked a very fair question: “Okay, but where exactly do we configure this MTU? On the physical switch? On the VDS? Everywhere?” That is one of the most common questions I hear, so this post answers it in plain language, with diagrams and tables, and then adds the technical detail for the admins who configure it.
TL;DR
- MTU is the biggest packet size a network link will carry. NSX overlay traffic needs bigger packets, so VCF needs a bigger MTU than the old default of 1500.
- The golden rule: every hop along the path must support the same (or larger) MTU. One device set too small silently drops the big packets.
- Minimum to deploy VCF: 1600. Recommended minimum: 1700. Best for performance: 9000 (jumbo frames) end to end.
- Where you set it: physical switches and routers, the vSphere Distributed Switch (VDS), VMkernel adapters, the NSX TEP / Global Fabric Settings, the NSX Uplink Profile, and (optionally) the guest VM.
- Who owns what: the physical switches are the network team’s job; the VDS, VMkernel and NSX values are the VMware admin’s job. They must agree on the number.
- What is MTU, in plain English
- Why VCF and NSX need a bigger MTU
- The golden rule: every hop must match
- Where exactly do you configure MTU? (the master table)
- Which value should I use? (cheat sheet)
- The full picture, layer by layer
- Where to click: a quick configuration map
- How to test that it actually works
- Common mistakes that cause outages
- FAQ
- Key takeaways
1. What is MTU, in plain English Basics
2. Why VCF and NSX Need a Bigger MTU Overlay
3. The Golden Rule: Every Hop Must Match Critical
This is the single most important idea, and the answer to the customer’s question. MTU is not set in one place. A packet travels through many devices, and every device along the path must support the chosen MTU. If even one link is set smaller, the big packets are dropped there, and you get strange, hard-to-diagnose failures (vSAN errors, vMotion failures, NSX tunnels down) while small pings still work fine.
4. Where Exactly Do You Configure MTU? The answer
Here is the direct answer to “where all do we need to set this?” Read it as a checklist. Each row is one place MTU lives, what it means in plain terms, the value to use, and who normally owns it.
| Where (the layer) | What it is, simply | Recommended MTU | Who configures it |
|---|---|---|---|
| Physical switches and routers (the underlay) | The real network gear your hosts plug into, including any router or spine between racks. | 9000 (set highest) | Network / customer team |
| LACP / port-channel (if used) | Bonded physical switch ports. | 9000 | Network team |
| vSphere Distributed Switch (VDS) | The shared virtual switch all hosts in the cluster use. | 9000 | VMware admin (in vCenter) |
| VMkernel adapters (vMotion, vSAN, etc.) | The host’s own interfaces for services like live migration and storage. | 9000 for vSAN and vMotion | VMware admin |
| NSX TEP / Global Fabric Settings (Global Tunnel Endpoint MTU) | The MTU for NSX overlay tunnels between hosts. This is the main NSX value. | 1700 default, 9000 optimal | NSX / VMware admin |
| NSX Uplink Profile MTU | Per-profile overlay MTU (used with N-VDS host switches). | Match the TEP value | NSX admin |
| NSX Gateway interface MTU (Global Logical Interface / Tier-0) | MTU on the router uplinks leaving NSX toward the physical network. | 1500 default; keep at least 200 below fabric | NSX admin |
| Guest VM (OS inside the VM) | The MTU set inside Windows or Linux running in the VM. | 1500 (up to 8800 only if fabric is 9000) | Application / VM owner |
5. Which Value Should I Use? Cheat sheet
| Value | Meaning | When to use |
|---|---|---|
| 1600 | The absolute minimum to deploy VCF (covers the overlay header). | Bare minimum only; not recommended long term. |
| 1700 | Recommended minimum. Adds headroom for future header growth. | When the physical network cannot do jumbo frames. |
| 9000 | Jumbo frames. Best throughput for vSAN, vMotion and overlay. | Recommended whenever the physical underlay supports 9000. |
| 1500 | The classic default, used inside guest VMs. | Leave guest VMs at 1500 unless you have a specific need. |
6. The Full Picture, Layer by Layer Map
7. Where to Click: A Quick Configuration Map How-to
| Setting | Where to find it |
|---|---|
| VDS MTU | vCenter > Networking > select the Distributed Switch > Edit Settings > Advanced > MTU |
| VMkernel MTU | vCenter > Host > Configure > VMkernel adapters > edit adapter > MTU (or it follows the VDS) |
| NSX Global TEP MTU | NSX Manager > System > Settings > Fabric Settings > Global Fabric Settings (default 1700) |
| NSX Uplink Profile MTU | NSX Manager > System > Fabric > Profiles > Uplink Profiles |
| NSX Gateway interface MTU | NSX Manager > Networking > Global Networking Config (or the Tier-0 uplink interface) |
| Physical switch MTU | On the switch itself, by the network team (per port or global, plus any L3 interfaces) |
8. How to Test That It Actually Works Verify
Never assume MTU is correct because the config looks right. Test it with a “do not fragment” ping from an ESX host. If a big packet gets through without being broken up, the whole path supports that size.
For a 9000 MTU path, send a 8972-byte payload (9000 minus 28 bytes of IP and ICMP overhead) with the do-not-fragment flag:
vmkping -I vmk0 -d -s 8972 <destination_ip>
For a 1600 MTU path, test with -s 1572. If the ping succeeds, the path supports jumbo frames. If it fails or times out while a normal ping works, one hop is set too small.
9. Common Mistakes That Cause Outages Avoid
| Mistake | What you see | Fix |
|---|---|---|
| VDS set to 9000 but physical switch still 1500 | vSAN errors, vMotion fails, NSX tunnels down, but small pings work | Set physical switches to 9000 first |
| Forgetting the router or spine between racks | Works within a rack, breaks across racks (L3) | Set MTU on the L3 interfaces too |
| Guest VM set to 9000 on a 1600 fabric | Large file transfers stall; small traffic fine | Keep guests 200+ bytes below fabric |
| Only configuring one side of a link | Intermittent, one-directional drops | Both switch ports must match |
| Assuming config equals working | Surprise failures under load | Always verify with vmkping -d -s |
10. FAQ
Do I really need jumbo frames (9000), or is 1600 enough?
1600 is the minimum just to deploy VCF. It works, but you lose performance. For production, especially with vSAN and vMotion, 9000 is strongly recommended when your physical network supports it. If it cannot, use 1700 as a safe minimum.
What actually happens if the MTU does not match across devices?
The device with the smaller MTU receives a packet that is too big and simply drops it (when the do-not-fragment bit is set, which overlay traffic uses). Small packets still pass, so basic pings succeed while real workloads fail. This is why MTU issues are so confusing to troubleshoot.
Why does VMware say 1700 instead of exactly 1600?
1600 just barely covers today’s overlay header. 1700 adds headroom so future increases in the Geneve header will not break you. It costs nothing extra to use 1700, so it is the safer default.
Does the management network also need 9000?
No. Management traffic is fine at 1500. Jumbo frames matter most for vSAN, vMotion, and NSX overlay (TEP) traffic. Many designs keep management at 1500 and set the data and overlay networks to 9000.
As the customer, what is my part versus the VMware admin’s part?
The physical switches and routers are your network team’s responsibility. The VDS, VMkernel, and NSX values are the VMware/NSX admin’s responsibility. The single thing both sides must do is agree on the same MTU number end to end.
How do I check the MTU currently set on a switch or host?
On ESXi, run esxcli network vswitch dvs vmware list for the VDS MTU, or check the VMkernel adapters in vCenter. On a physical switch, the network team can show the port and interface MTU. Then confirm end to end with a do-not-fragment ping.
11. Key Takeaways
MTU is just the maximum packet size a link will carry, and NSX in VCF needs more than the old 1500 default because the overlay adds an envelope of about 100 bytes. The answer to “where do we set it” is: in several places that must all agree. The physical switches and routers (owned by the network team) should be set highest, typically 9000. The VDS, the VMkernel adapters, and the NSX overlay (TEP / Uplink Profile) should match, with 1600 as the bare minimum, 1700 as a safe minimum, and 9000 for best performance. Keep guest VMs and gateway interfaces at least 200 bytes below the fabric. Then verify the whole path with a do-not-fragment ping rather than trusting the config screen. Get those pieces aligned and your VCF 9 networking will be fast and stable.
Primary reference: Broadcom, Guidance to Set Maximum Transmission Unit (VCF 9.0).
Written from a real VCF 9 deployment workshop question, to help non-technical stakeholders and new admins understand where MTU is configured and why.
About The Author
DrJha