TL;DR · Key Takeaways
- A stalled 9.1 upgrade is almost always one component stuck on one step, not the whole stack. Read the exact failed task name first, because ‘Import legacy components’, ‘Cloud Proxy Registration’ and a precheck stop each have a different fix.
- Retry is not free. For the legacy-components import stall the Retry button reuses the same NULL payload and fails again, so stop after the second try and switch to the Fleet lifecycle API.
- There is no single rollback button for VCF. vCenter and ESX lean on snapshots and the temporary-network switchover, SDDC Manager and NSX lean on file backups to external SFTP, and once VCF Management Services own the inventory you go forward, not back.
- Budget about 60 to 90 minutes to recover a typical import stall by hand: roughly 25 minutes to deploy the missing Salt and Telemetry components through the API and about 10 minutes to re-import each component.
- When you replace a cached SSL thumbprint or fix a stale Cloud Proxy mapping, trigger an inventory sync before you retry, or SDDC Manager keeps validating against the old cached value.
- Keep a recent SDDC Manager backup on external SFTP and a pre-upgrade vCenter snapshot. Those two are your real rollback path for the riskiest steps.
By the time you reach this part, the upgrade has mostly gone to plan, or it hasn’t. Either way, a VCF 9.1 upgrade that stalls looks scarier than it is. The task view shows a red Failed, the maintenance window is ticking, and the instinct is to hit Retry and hope. That instinct is usually the wrong one.
I’ve recovered enough of these to know the shape of it. A stall is almost always one component stuck on one thing, and the fix is specific to that thing. This part walks through how I read the failure, the three stalls I hit most and how to clear them, and where a rollback is genuinely possible versus where you’re committed to finishing the road.
Read the failed task before you touch anything
The single most useful thing on the screen is the name of the task that failed. VCF 9.1 orchestrates the upgrade as a chain of named steps, and the name tells you which subsystem is stuck. ‘Install VCF Operations – Import legacy components using SDDC lifecycle’ is a fleet inventory problem. ‘VCF Instance Cloud Proxy Registration validation failed’ is an integrations problem. A red precheck is a readiness problem. Same red banner, three different rooms to walk into.
So before you retry anything, copy the failed task name and expand its subtasks. Then decide which of the three paths below you’re on. Retrying blind wastes a chunk of your window and, in one common case, actively makes the state harder to reason about.
Where 9.1 upgrades actually stall
Most of the pain in a 9.1 upgrade clusters in the early, new-in-9.1 phases: the VCF Operations import and the VCF Management Services deployment. That makes sense. Those steps are where the fleet inventory migrates and the new management layer takes ownership, and they depend on DNS, the Cloud Proxy, and clean 9.0.x inventory data. By the time you’re into NSX, vCenter and ESX, you’re on the well-trodden lifecycle path that VCF has run for years.
The chart below is an illustrative tally from upgrades I’ve watched, not a vendor statistic, but the shape matches what the public KBs describe: the import and the management-services deploy are where you’re most likely to get stuck.
The practical read: put your attention and your recovery time on the first two phases. Table 1 shows what a hands-on recovery of the most common one actually costs in minutes, which is worth knowing before you promise a rollback time to a change board.
| Recovery step (import stall) | Typical time | Note |
|---|---|---|
| Two failed Retry attempts | 30 min | Wasted. Same NULL payload each time. |
| Deploy Salt RaaS, Salt Master, Telemetry via API | 25 min | Fleet lifecycle API, token then install calls. |
| Re-import VCF Automation in the Ops UI | 10 min | Add Component, enter primary VIP. |
| Re-import VCF Operations for Networks | 10 min | Add Component, platform node IP. |
| Verify components and trigger inventory sync | 15 min | Confirm all four show deployed. |
| Total hands-on recovery | ~90 min | Roughly 60 min if you skip the wasted retries. |
The three stalls I see most, and their fixes
Stall at import legacy components
This is the worked example, and it’s the one that punishes blind retries. During the VCF Operations upgrade the process stops at ‘Install VCF Operations – Import legacy components using SDDC lifecycle’. Under the hood the Fleet lifecycle is trying to import your 9.0.x inventory, and it hits NULL infrastructure properties, usually DNS and NTP details, on a component that was removed and re-added to the fleet at some point. Because those values are NULL, the import of the VCF Operations component details fails. Retrying reuses the same payload, so it fails the same way.
Seen this go wrong: a customer hit this at about 40 minutes into the Ops import, retried twice on the assumption it was a transient timeout, and burned half an hour before we read the subtask detail and saw the NULL property. The Salt RaaS, Salt Master and Telemetry components had not installed, and those are required before VMware Aria Automation can upgrade. The retries were never going to work.
The fix has four moves. First, in the VCF Operations UI go to Build > Lifecycle > VCF Management > Components and compare what’s deployed against your 9.0.x list to confirm which components are missing. Second, use the Fleet lifecycle API to deploy the missing ones (Salt RaaS, Salt Master, Telemetry, and the License Server): fetch an identity token, fetch the SDDC LCM ID, then fire the install calls. Third, clear the legacy Aria Automation 9.0 environment from vSphere. Fourth, re-import Automation and Operations for Networks through Add Component in the Ops UI so the fleet inventory is complete and the 9.1 upgrade can carry on.
Cloud Proxy Registration validation failed
This one shows up during the VCF Management Services deployment. SDDC Manager reports ‘VCF Instance Cloud Proxy Registration validation failed’, and if you try to add the VCF adapter by hand in VCF Operations you get ‘Failed to create AI resource. Resource with same key already exists’. The cause is that the VCF Instance adapter is not tied to a live Unified Cloud Proxy, and there’s a stale integration entry sitting in the database. Adding a fresh one collides with it.
Fix it by editing, not adding. In VCF Operations go to Administration > Integrations, find the VMware Cloud Foundation adapter instance, Edit it, and set the Collector/Group to the live Unified Cloud Proxy. Do the same for every workload domain and vCenter in that VCF instance. Confirm the Cloud Proxy shows Healthy and Online, then in Fleet Management > Lifecycle > VCF Management > Components trigger an inventory sync on the Operations component. Go back to SDDC Manager and retry the Management Services deployment. The validation passes once the mapping is real and synced.
Retry keeps failing on a certificate you already fixed
Here the upgrade stops validating the Operations for Networks certificate thumbprint. You replace the certificate on the appliance, click Retry in SDDC Manager, and it fails again with the old thumbprint. That’s not a stuck workflow, it’s a cache: SDDC Manager holds the SSL thumbprint in its lifecycle service, so Retry keeps comparing against the value it already had. The move is to force a rediscovery or inventory sync so the new thumbprint is read in, and only then retry. Until the cache refreshes, every retry checks stale data.
| Symptom | Root cause | Fix |
|---|---|---|
| Stalls at ‘Import legacy components using SDDC lifecycle’ | Fleet import hits NULL DNS/NTP properties on a re-added component; Retry reuses the same payload | Deploy Salt RaaS, Salt Master, Telemetry, License Server via Fleet lifecycle API, then re-import Automation and Ops for Networks in the UI |
| ‘Cloud Proxy Registration validation failed’; adapter add says ‘same key already exists’ | VCF adapter not bound to a live Unified Cloud Proxy; stale integration entry | Edit the VCF adapter Collector/Group to the live Cloud Proxy for every domain and vCenter, confirm Healthy, inventory sync, retry |
| Retry keeps failing on a certificate you already replaced | SDDC Manager caches the old SSL thumbprint in its lifecycle service | Force rediscovery or inventory sync so the new thumbprint loads, then retry |
| Automation upgrade fails during precheck | A blocking readiness item on the Automation component | Clear the flagged item, rerun the Automation precheck before resuming the plan |
What I’d do: give any stall two retries, no more. If the second retry lands on the same subtask with the same message, stop and read the KB for that exact task name. Three of the four failures above are cache or inventory problems that a retry cannot touch, and every extra retry is 10 to 15 minutes you don’t get back.
Rolling back, and where the real boundaries are
There is no single rollback button for VCF, and anyone who tells you otherwise hasn’t had to use one at 2 AM. What you have instead is a set of component-level safety nets, and they don’t all cover the same ground. vCenter and ESX are the reversible end of the stack because they lean on snapshots and, for vCenter, the temporary-network switchover that keeps the old appliance intact until you commit. SDDC Manager and NSX rely on file-based backups to external SFTP, which is a restore, not an instant undo. And VCF Management Services is the one-way door: once it owns the fleet inventory, your path is forward.
Before you start: a rollback is only as good as the backup behind it. Confirm a recent successful SDDC Manager backup on external SFTP, take a supported vCenter snapshot before its upgrade, and check that managed components have current backups. Do this inside a change window, and never treat any of these steps as risk-free. If the backup is stale, you don’t have a rollback, you have a hope.
The practical consequence is about ordering your own confidence. Everything before Management Services deploys is still cleanly recoverable. Once that deploy succeeds and the fleet inventory moves, treat the rest as a forward-only run and lean on your component backups if a later step fails. That’s why I hold the riskiest validation, and my slowest, most careful checking, right around the management-services boundary.
Signs it worked
After you clear a stall, don’t just watch the failed task flip to green and walk away. Confirm the recovery held before you resume the plan.
- All four fleet components (Salt RaaS, Salt Master, Telemetry, License Server) show deployed in Build > Lifecycle > VCF Management > Components.
- The Cloud Proxy shows Healthy and Online, and the inventory sync completes without error.
- The previously failed task now shows Successful, and the next task in the chain has started on its own.
- Re-imported components (Automation, Operations for Networks) appear in the inventory with their versions, not blank.
If any of those still look wrong, the stall isn’t cleared yet, whatever the task banner says. That is the whole game with a stalled upgrade: fix the actual state, not the status light.
Common questions
How many times should I click Retry before I stop?
Twice. A first retry catches a genuine transient timeout. A second confirms it isn’t one. If the second retry fails on the same subtask with the same message, the payload or the cached state is the problem, and a third attempt just spends your window.
Can I roll the whole platform back to 9.0.x if the upgrade goes badly?
Not as one action. You roll back per component using snapshots for vCenter and ESX and file backups for SDDC Manager and NSX. Once VCF Management Services has taken over the fleet inventory, that piece is forward-only, so your safety after that point is restoring individual components, not reverting the whole stack.
Why does my certificate fix not take effect on Retry?
Because SDDC Manager cached the old SSL thumbprint. Replacing the certificate on the appliance doesn’t refresh that cache on its own. Trigger a rediscovery or inventory sync so the new thumbprint is read, then retry.
The import stall needs API calls. Is there a UI-only way?
Not for this one. The retry path in the UI reuses the failing payload, so you deploy the missing Salt and Telemetry components through the Fleet lifecycle API, then finish the re-import from the Ops UI. Have an API client and the runtime FQDNs ready before the window if you think you might hit it.
Should I open a support case or self-recover?
If the failure matches a published KB exactly, like the import stall or the Cloud Proxy registration error, self-recovery is quick and documented. If the state looks corrupted, if inventory is inconsistent, or if you’re outside the documented fixes, open a case before you improvise on a production fleet.
« Previous: Part 13 | Complete Guide | This is the final part in the series.
References
- Broadcom KB 441127: Upgrade to VCF 9.1 stalls at import legacy components stage
- Broadcom KB 443737: VCF 9.1 upgrade fails with VCF Instance Cloud Proxy Registration validation failed
- Broadcom KB 440630: Upgrade sequence and related issues for VCF and vSphere Foundation 9.1
- Broadcom KB 440282: VCF 9.1 upgrade binaries do not show up in VCF Operations 9.0
- Broadcom KB 440407: Upgrading VCF Operations from 9.0.2 to 9.1 returns sync error when upgrading ESXi host
- Angry Admin: Upgrading VMware Cloud Foundation from 9.0.2 to 9.1, practical notes
About The Author
DrJha