TL;DR · Key Takeaways
- Validate top down: VCF Operations and Management Services first, then SDDC Manager, then NSX, vCenter, and finally ESX and vSAN. Budget roughly 115 minutes of checking for a two-domain fleet.
- Run the SoS health check from SDDC Manager as root: cd /opt/vmware/sddc-support then ./sos –health-check –domain-name ALL –skip-cert-check. You want green across connectivity, compute, and storage.
- Do not trust one screen. Confirm the same build number from the VCF lifecycle view and from each component UI before you believe it.
- Let vSAN resync drain to near zero before you call the upgrade done. A 3.8 TB backlog clearing at about 600 GB per hour takes roughly 6.5 hours.
- Only after every layer is green do you remove the upgrade snapshots and take fresh backups of the newly upgraded components.
By the time the last ESX host reboots on 9.1, it is tempting to close the change ticket and go to bed. I have learned to slow down right here. The upgrade wizard turning green means the workflow finished. It does not mean the platform is healthy. Those are two different claims, and the gap between them is where weekend incidents live.
This part is the checklist I actually run after a VCF 9.1 upgrade. I validate in layers, starting at VCF Operations and working down to vSAN. I run the SoS health check from SDDC Manager. I only remove snapshots and take fresh backups once every layer agrees on the same story. It is a boring hour, and it is the hour that keeps a clean upgrade clean.
Validate in layers, from VCF Operations down to the hosts
In the 9.x model, VCF Operations is the control point for lifecycle, and Management Services owns the fleet inventory. So validation starts at the top. If Operations or Management Services is unhappy, every version number it reports for the layers below is suspect. Work down the stack in the same order you upgraded it, and treat each layer as a gate: prove it healthy before you trust the next one.
Start at VCF Operations and Management Services
Log in to VCF Operations and confirm the cluster status is online with every node online. During the upgrade the cluster flips offline and back, so a single node stuck in going-online is a real finding, not a cosmetic one. Then check that the old fleet management appliance is gone. In 9.1 the migration decommissions and powers off the previous appliance, and a leftover powered-on appliance means the migration did not finish cleanly. Confirm the cloud proxy shows healthy, since Operations upgrades it as part of the cluster upgrade and a stale proxy quietly breaks data collection.
SDDC Manager and the lifecycle view
Open the lifecycle view in VCF Operations and read the reported versions for every managed component. This is your single inventory of record, and it should show 9.1 builds where you expect them and nothing stuck mid-transition. Cross-check that the depot connection is still active, because binary management and any follow-on patching depend on it. If the lifecycle view and a component UI disagree about a version, believe neither until you reconcile them.
NSX, vCenter, then ESX and vSAN
For NSX I am deliberately strict. Check the management plane, the control plane, transport node status, edge node status, and routing, both from the lifecycle view and from the NSX Manager UI. For vCenter, confirm the build from the VCF lifecycle view and from the vCenter UI, and make sure services are up after the temporary-network switchover the upgrade uses. For ESX, confirm host builds and cluster image compliance. Then finish at vSAN, which is usually the slowest thing to settle because resync runs in the background well after the hosts report done.
Here is the same budget as a table. These are the numbers I plan a change window around, not a promise. A big vSAN cluster can push that last row well past 45 minutes.
| Layer | Where you check | What green looks like | Time |
|---|---|---|---|
| VCF Operations | Operations UI, cluster status | All nodes online, old fleet appliance gone, cloud proxy healthy | 10 min |
| SDDC Manager | Lifecycle view, SoS health check | Expected 9.1 builds, depot active, SoS all green | 15 min |
| NSX | NSX Manager UI and lifecycle view | Manager, control plane, transport nodes, edges, routing all up | 30 min |
| vCenter | vCenter UI and lifecycle view | Build matches, services up after switchover | 15 min |
| ESX and vSAN | vCenter, vSAN health, cluster compliance | Host builds correct, image compliant, resync near zero | 45 min |
Table 1. A rough validation budget of about 115 minutes for a two-domain fleet.
Run the SoS health check and read it properly
The Supportability and Serviceability tool, SoS, is the fastest way to get one honest opinion across connectivity, compute, and storage. It runs on the SDDC Manager appliance. SSH in as the vcf user, become root, then run it against every domain.
cd /opt/vmware/sddc-support
./sos –health-check –domain-name ALL –skip-cert-check
Read every section, not just the summary line. SoS separates connectivity, compute, storage, and certificate results, and a stack can be fine at the compute layer while a certificate is days from expiry. One quirk worth knowing: the SoS API has a guardrail that complains if a workflow is still running. Right after an upgrade a trailing task can trip it, so let the fleet settle for a few minutes and re-run rather than chasing a false red. If a specific domain looks messy, drop –domain-name ALL and target that one domain so the output is easier to read.
A worked example: draining the vSAN resync backlog
On a recent management-plus-one-workload-domain upgrade, every ESX host in the workload domain reported 9.1 and green within the maintenance window. The cluster summary, though, showed 1,240 objects still resyncing and about 3.8 TB of data in flight. That is normal after rolling host upgrades with vSAN, because each host entering and leaving maintenance mode triggers rebuilds. It is also exactly the moment people declare victory too early.
I measured the drain rate at roughly 600 GB per hour on that cluster, so the backlog needed about 6.5 hours to reach near zero. I did not remove a single snapshot or close the ticket until the resync objects were down to a handful and the vSAN health check was green. If a second host had failed while the cluster was still rebuilding from the first, I would have been one fault away from data unavailability. The picture below is the actual drain curve I tracked.
Read the whole fleet at a glance
Once you have upgraded more than one domain, a single grid of statuses saves you from tab-hopping. I sketch a quick matrix of domains against components and colour it in as each check passes. I want to catch the one amber cell that a per-domain view would let you skim past. In the example below, workload domain two still has an amber vSAN cell because resync had not finished, which is the signal to wait rather than sign off.
When a check fails, remediate before you move on
A failed check is not a reason to panic and it is not a reason to push forward. It is a reason to stop, fix, and re-run that check. The riskiest habit I see is treating a red or amber result as a formality and continuing anyway, because the fleet inventory drifts a little further out of sync with every step you take on top of a broken layer.
These are the failures I hit most often after a 9.1 upgrade, what they look like, and how I clear them.
| What you see | Likely signal | What I do |
|---|---|---|
| Old fleet appliance still powered on | Migration to Operations did not finish | Confirm the new services own inventory, then decommission the old appliance per the guide |
| Cloud proxy shows disconnected | Proxy did not follow the cluster upgrade | Restart or re-register the proxy, confirm data collection resumes |
| Lifecycle view and UI show different builds | Inventory refresh lag or stuck task | Let trailing tasks finish, refresh inventory, re-read both screens |
| SoS certificate section amber | A component cert near expiry | Rotate the cert before it lapses, do not defer it past the change |
| vSAN health amber, objects resyncing | Normal post-upgrade rebuild in progress | Wait for the backlog to drain to near zero, keep hosts out of maintenance |
Table 2. Common post-upgrade findings and the remediation I reach for.
Field note and verdict
Signs it worked
You know the upgrade truly landed when the VCF Operations cluster is online with the old fleet appliance decommissioned, the lifecycle view shows the expected 9.1 builds with no component stuck mid-transition, SoS reports green across connectivity, compute, storage, and certificates for every domain, NSX management and edges are up with routing intact, vCenter and ESX builds match between the lifecycle view and each UI, and vSAN health is green with resync at near zero. At that point you remove the snapshots, take fresh backups, and close the ticket with a clear conscience.
Common questions
Is the SoS health check enough on its own?
No. SoS is the best single first pass across connectivity, compute, and storage, and I always run it. It is still a summary. It will not catch a vCenter service that is running but erroring, or an NSX edge that is up while dropping a routing session. Pair it with a look at each component UI.
How long should I wait on vSAN before signing off?
Until resync objects are down to a handful and vSAN health is green. Estimate it from the backlog and the drain rate. A 3.8 TB backlog at about 600 GB per hour is roughly 6.5 hours. Keep hosts out of maintenance mode while it drains so you do not restart the rebuild.
When can I delete the upgrade snapshots?
Only after every layer is green. Snapshots are your quickest rollback while validation is still running. Delete them once health is proven, then take fresh backups of the upgraded components to external storage. Do not leave old snapshots sitting around either, since they waste storage and hurt performance.
The lifecycle view and a component UI disagree on the version. Which is right?
Trust neither until you reconcile them. Usually a trailing task or an inventory refresh lag is to blame. Let running workflows finish, refresh the inventory, and re-read both screens. If they still disagree, that is a support case, not something to build more upgrade steps on top of.
Do I really need to check certificates during a health pass?
Yes. SoS surfaces certificate state, and an upgrade is a natural moment for a cert to be close to expiry. Rotating a cert that is about to lapse during your change window is far cheaper than a surprise outage a week later.
This series covers upgrade procedures for infrastructure teams. Any production-affecting step should run inside a change window with backups confirmed and a rollback path agreed. Validate against your own environment before acting.
References
- How to Upgrade to VMware Cloud Foundation 9.1, VCF Blog
- Perform the Upgrade to VCF Operations 9.1, Broadcom TechDocs
- Upgrading VMware Cloud Foundation from 9.0.2 to 9.1, Angry Admin
About The Author
DrJha