What it does: Removes a failed ESXi host from its cluster, decommissions it, then commissions and adds a healthy replacement, keeping the cluster and its vSAN datastore in service throughout.
Depends on: a workload domain with a failed host, spare redundancy so the cluster still tolerates the loss, a compatible replacement host, and an existing network pool.
- Host replacement is remediation. Run it when a host has failed, not when you want to grow a cluster with a healthy host.
- Work spans three consoles: VCF Operations for the alert and fleet health, the SDDC Manager UI to remove the host from the cluster, and the vSphere Client to decommission and commission hosts.
- If the failed host held vSAN capacity, vSAN starts rebuilding absent components after the object repair timer expires, which defaults to 60 minutes, so confirm the cluster still meets its failures to tolerate setting.
- Sequence: detect, evacuate roles, remove from the cluster, decommission, reimage, commission the replacement, add it back.
- A host that was part of a VCF domain must be reimaged before you commission it again.
- The replacement host must match the cluster ESX build, sit on a network pool, and pass thumbprint validation before it commissions.
This part shows you how to replace a failed ESXi host in a VCF 9.1 cluster while keeping the rest of the cluster running. You will confirm the failure, move any roles and workloads off the host, remove it from the cluster in the SDDC Manager UI, decommission it from the global inventory in the vSphere Client, then reimage, commission and add a healthy replacement. Each step names the exact console so you always know where you are working.
Treat this as a scenario you run on demand. A failed host means one that vCenter reports as not responding, disconnected, or repeatedly crashing, or one whose vSAN disks have failed. If you simply want more capacity, add a healthy host instead, which is a different and lower risk task. For the operating model that sits behind these tasks, see the Day-2 operations overview.
Prerequisites
| Requirement | Detail |
|---|---|
| Replacement host | A server on the Broadcom compatibility guide, with a disk layout that matches the vSAN configuration of the failed host. |
| ESX build | Same ESX version and image the cluster runs, so the host passes cluster image compliance after it joins. |
| Network pool | An existing network pool with free management, vMotion and vSAN addresses for the new host. |
| Access | SDDC Manager UI administrator, plus a vSphere Client session with management domain vCenter access or VCF single sign on with vCenter linking. |
| Redundancy | The cluster still meets its failures to tolerate value with the failed host absent, so removing it does not risk data. |
| Backups | Current SDDC Manager, vCenter and NSX Manager backups, in case remediation needs a rollback. |
Step 1, Confirm the failure and its impact
Before you remove anything, confirm the host has genuinely failed and check what its loss does to redundancy. VCF Operations raises the alert, and the vSphere Client shows the host state and vSAN health.
- In VCF Operations, open the Alerts view and read the alert for the affected host to see the reported fault.
- In the vSphere Client, select the cluster and open the Monitor tab, then check the host connection state under Hosts.
- On the same Monitor tab, open vSAN and then Skyline Health to see whether objects are degraded or resyncing.
- Note the cluster Failures to tolerate value under the vSAN storage policy so you know how much redundancy remains.
- Record the host FQDN, management IP and service tag so you can match them when you image the replacement.
If vSAN reports that the cluster no longer meets its policy, wait for any in progress resync to finish, or add temporary capacity, before you remove the host. Removing a host while the cluster is below policy can mark the datastore read only or lose data.
Step 2, Evacuate roles and data from the failed host
Move anything that still lives on the host so its removal is clean. If the host is reachable, put it into maintenance mode with a vSAN data migration option. If the host is dead, vSAN handles rebuild on its own once the object repair timer expires.
- In the vSphere Client, check whether the host runs an NSX Edge node, and if it does, redeploy or move that Edge node to another host in the same cluster first.
- Move or delete any virtual machines that were created outside VMware Cloud Foundation and still sit on the host.
- Right-click the host and select Maintenance Mode, then Enter Maintenance Mode.
- Under vSAN data migration, choose Full data evacuation when the host is healthy enough to copy its data, or Ensure accessibility when you only need remaining copies to stay reachable.
- Click OK and wait for the host to enter maintenance mode.
| Data migration option | What it does | When to use |
|---|---|---|
| Full data evacuation | Copies all vSAN components off the host before it enters maintenance mode. | Planned replacement of a still healthy host, with capacity to hold the data. |
| Ensure accessibility | Moves only enough components to keep virtual machines accessible. | Faster option when the cluster still holds another full copy of the data. |
| No data evacuation | Moves no vSAN data off the host. | Host already failed and offline, so vSAN rebuilds absent components elsewhere. |
Step 3, Remove the failed host from the cluster
Remove the host from the vSphere cluster through the SDDC Manager UI, which keeps the VCF inventory in step with vCenter. This runs in the SDDC Manager UI, not the vSphere Client.
- Open the SDDC Manager UI at
https://sddc-manager-fqdnand sign in as an administrator. - In the navigation pane, click Inventory, then Workload Domains.
- Click the name of the workload domain that contains the failed host.
- Click the Clusters tab, then click the name of the affected cluster.
- Click the Hosts tab, select the failed host, and click Remove Selected Hosts.
- If removal drops the cluster below its minimum host count, click Force Remove.
- Click Remove to confirm, and wait for the host to leave the hosts table.
Force Remove exists because a three host vSAN cluster cannot normally drop below three hosts. Use it only when you will add the replacement straight after, and only once you have confirmed remaining redundancy in Step 1.
Step 4, Decommission the failed host
Removing a host from the global inventory is called decommissioning. In VCF 9.1 this moved to the vSphere Client, under the global inventory host list.
- In the vSphere Client, click Global Inventory Lists, then Hosts.
- Click the Unassigned Hosts tab, where the removed host now appears.
- Select the failed host and click Decommission Selected Hosts.
- Leave Skip failed hosts during decommissioning enabled so a stuck host does not block the task.
- Click Confirm and wait for the host to leave the unassigned list.
A host that was part of a VCF domain must be reimaged before you commission it again. That reimage happens in the next step, and it is why a failed physical host and a fresh chassis follow the same path from here.
Step 5, Prepare and reimage the replacement host
Bring the replacement chassis to the same ESX build the cluster runs, then confirm its identity before commissioning. Matching the build keeps the host image compliant once it joins.
- Install the ESX build that matches the cluster on the replacement host, following the imaging approach in the VCF 9.1 deployment guide.
- Set the host management IP, subnet, gateway, DNS and NTP to the values you recorded in Step 1.
- Set a known root password, since commissioning asks for the host root credentials.
- Confirm each physical disk that vSAN will claim is empty and shows no stale partitions from prior use.
- From the host, confirm the build with a shell command so it matches the cluster.
esxcli system version get
# Product: VMware ESXi
# Version: 9.0.0
# Build: Releasebuild XXXXXXXX
# match this Build to the value the cluster reportsIf your replacement adds capacity as well as restoring redundancy, review the disk and storage guidance in Part 15 so the new disks land in the right vSAN ESA configuration.
Step 6, Commission the replacement host
Commissioning adds the host to the global inventory so VCF can consume it. In VCF 9.1 this runs from the global inventory host list in the vSphere Client.
- In the vSphere Client, click Global Inventory Lists, then Hosts.
- Click the Unassigned Hosts tab, then click Commission Hosts.
- Review the host commission checklist and click Proceed.
- Keep Add new selected, then enter the host FQDN, choose the Storage Type that matches the cluster, pick the network pool, and enter the root credentials.
- Click Confirm all Finger Prints, then click Validate All to check the host thumbprint.
- Review the validated host details and click Commission.
- Wait for the configuration status to change to Active.
For a vSAN cluster, set the storage type to vSAN so the host joins with the right disk role. A validation error at this point usually means DNS, the root password or the thumbprint does not match, all of which the common errors table below covers.
Step 7, Add the replacement host back to the cluster
With the host commissioned and Active, add it to the cluster so the cluster returns to its full host count. Adding a host to an existing cluster is owned by Part 12, so follow those steps and return here to verify.
- In the SDDC Manager UI, click Inventory, then Workload Domains, and open the affected domain.
- Click the Clusters tab, click the cluster name, then start the add host action described in Part 12.
- Select the commissioned replacement host from the available hosts and submit the task.
- Watch the task to completion, then take the host out of maintenance mode if it is still in it.
Once the host joins, vSAN rebalances components across the restored member count. Let that resync settle before you call the replacement done, which the verification below confirms.
How to verify the replacement
Confirm the cluster is back to full strength across three consoles before you close the task.
- In the SDDC Manager UI, open the cluster Hosts tab and confirm the replacement host is listed and active.
- In the vSphere Client, open the cluster Monitor tab, then vSAN and Skyline Health, and confirm all checks are green with no resync remaining.
- In VCF Operations, confirm the host reports healthy and the earlier alert has cleared.
- Confirm the host image shows as compliant so the cluster is not carrying drift.
You can confirm no vSAN components are still rebuilding with a short PowerCLI check against the cluster.
Connect-VIServer -Server vcenter-fqdn
Get-VsanResyncingComponent -Cluster "wld01-cluster01"
# an empty result means resync is complete
Get-VMHost -Name "esx07.example.local" | Select Name, ConnectionState, PowerStateCommon errors and fixes
| Symptom | Likely cause | Fix |
|---|---|---|
| Remove host is blocked | The host still runs an NSX Edge node, or holds workload virtual machines. | Move the Edge node to another host in the cluster and move or delete non VCF virtual machines, then retry. |
| Commission fails validation with an SSL trust error | The host certificate is not trusted by SDDC Manager, or the thumbprint was not confirmed. | Confirm all fingerprints, validate again, and regenerate the host certificate if the trust error persists. |
| Commission fails on network pool or DNS | Forward or reverse DNS is missing, or the network pool has no free addresses. | Fix forward and reverse DNS for the host FQDN and confirm the network pool has free vMotion and vSAN addresses. |
| vSAN health stays degraded after the host joins | Resync is still running, or a replacement disk was not claimed. | Wait for resync to finish, then claim any unclaimed disks for vSAN so the host contributes full capacity. |
Common questions
Do I have to wait for vSAN to rebuild before removing a dead host? No. When a host is offline, vSAN starts rebuilding absent components once the object repair timer expires, which defaults to 60 minutes. You can remove the host from the cluster once you confirm remaining copies keep the datastore compliant.
Can I reuse the same physical host after a transient failure? Yes, but a host that was part of a VCF domain must be reimaged before you commission it again. Treat a recovered chassis the same as a new one from Step 5 onward.
Is host replacement the same as adding a host? No. Replacement is remediation for a failed host and includes removal and decommissioning first. Adding a healthy host to grow a cluster skips those and is covered in Part 12.
Which console commissions hosts in VCF 9.1? The vSphere Client, under Global Inventory Lists and Hosts. In earlier releases this ran from SDDC Manager, so older guides point to a different place.
References
- Broadcom TechDocs, Decommission ESX Hosts (VCF 9.1)
- Broadcom TechDocs, Commission ESX Hosts (VCF 9.1)
- Broadcom TechDocs, Remove a Host from a vSphere Cluster in a Workload Domain
- Broadcom TechDocs, Managing ESX Hosts in VMware Cloud Foundation
About The Author
