, ,

Upgrade SDDC Manager and the Management Domain to VCF 9.1 (VCF 9.1 Upgrade Series, Part 11)

How to upgrade SDDC Manager from VCF Operations and then bring the VCF 9.1 management domain core up in order: NSX Manager, vCenter, ESX, and the NSX Edge cluster, with prechecks, timings and the depot gotcha.

VCF 9.1 Upgrade · Part 11 of 14

TL;DR · Key Takeaways

  • SDDC Manager is not upgraded from its own UI anymore. From 9.0.x you drive it from VCF Operations under Build, Lifecycle, VCF Instances, then the SDDC Manager Updates tab, Update Now.
  • Do four things before you click: a fresh SFTP backup of SDDC Manager, a snapshot of the appliance, backups of the components it manages, and a clean precheck. Skip the snapshot and you have no fast way back.
  • After SDDC Manager reaches 9.1, its depot connection breaks until you re-point it at an activation code from the VCF Business Services console instead of the old download token.
  • The management domain core order is fixed: NSX Manager, then vCenter, then ESX hosts, then the NSX Edge cluster last to finalize NSX.
  • Pick optimized or sequential for vCenter and NSX Manager in the domain upgrade plan. Optimized is faster but stages a new vCenter and does a switchover, sequential is simpler to reason about.
  • vSphere Lifecycle Manager baselines are gone in 9.x. Any baseline cluster must move to a vLCM image before ESX can go to 9.1.

By this point VCF Operations is on 9.1 and the dependency components from Part 10 are validated. SDDC Manager is the next appliance to move, and in the real upgrade order it reaches 9.1 before VCF Management Services is deployed. I covered the Management Services deployment back in Part 9 because it belongs with the depot and licensing story, but on the day itself you upgrade SDDC Manager first. This part is the SDDC Manager step and the management domain core that follows.

The part that trips people is that the workflow moved. In earlier VCF you logged into SDDC Manager and clicked through its own Updates tab. In 9.1 the fleet is driven from VCF Operations, and SDDC Manager is just one of the instances you upgrade from there. I will walk through the SDDC Manager step, the depot gotcha that follows it, and then the management domain core components in the order that actually works.

Where this step sits in the upgrade

The 9.1 upgrade is a chain, and Part 11 is the middle links. VCF Operations is already on 9.1. SDDC Manager is next and must reach 9.1 before VCF Management Services is deployed, then you work through the rest of the management domain core: NSX Manager, vCenter, ESX, and the NSX Edge cluster. Workload domains are a separate job you can defer, and I cover those in Part 12.

VCF OperationsdoneMgmt servicesdoneSDDC Managerthis partMgmt domainthis partWorkloadPart 12
Figure 1. Part 11 covers SDDC Manager and the management domain core. Workload domains are deferred to Part 12.

Upgrade SDDC Manager first

SDDC Manager has to reach 9.1 before you can plan the rest of the management domain. The binary is a normal upgrade bundle, but from a 9.0.x fleet you no longer start it from the SDDC Manager UI. You start it from VCF Operations, where the fleet lifecycle now lives.

Before you start

Four things, and none of them are optional in a production window. Take a fresh backup of SDDC Manager to an external SFTP server. Take a snapshot of the SDDC Manager appliance so you have a fast rollback if the update stalls. Confirm you have recent successful backups of the components SDDC Manager manages. If you run the VMware Avi Load Balancer, check the Product Interoperability Matrix so its version is compatible with the SDDC Manager target before you move. Then run the upgrade precheck and clear whatever it flags.

Before you start: the appliance snapshot is the one people skip because it feels redundant next to the SFTP backup. It is not. A restore from SFTP is a rebuild-and-restore, a snapshot revert is minutes. During an upgrade window you want the minutes option.

Run the update from VCF Operations

From a 9.0.x fleet the click path is short. Log in to VCF Operations. From the top navigation click Build. In the left pane click Lifecycle. Expand VCF Instances and select the instance that hosts the SDDC Manager you want to upgrade. Open the SDDC Manager Updates tab, click the available update, and click Update Now. VCF Operations handles the rest and you monitor it from the same screen.

Reconnect the depot with an activation code

Here is the step that quietly breaks the next download if you miss it. Once SDDC Manager is on 9.1, go to Administration, then Depot Settings, and reconfigure your online or offline depot to authenticate with an activation code from the VCF Business Services console. The old 9.0.x download token no longer applies. If you plan the management domain upgrade before fixing this, the plan cannot pull bundles and you lose time chasing a permissions ghost that is really just stale depot auth.

Table 1. SDDC Manager precheck flags and how I clear them
Precheck flagWhy it failsFix before you proceed
Backup not recentNo successful SFTP backup inside the windowRun an on-demand backup and confirm it lands on the SFTP target
Password or certificate expiringA managed account or cert expires mid-upgradeRotate or extend it, then rerun the precheck
Avi version not compatibleLoad balancer sits below the supported floorCheck the interop matrix and bring Avi in line first
Disk space low on applianceLog or bundle buildup fills a partitionClear old bundles and logs, confirm free space, rerun

Plan the management domain upgrade

With SDDC Manager on 9.1 and the management services deployed, you plan the domain upgrade in VCF Operations. Planning is where you set the target VCF version, confirm the component versions, and define the sequence for the domain components. You also run prechecks again at the domain level, because a management domain has more moving parts than SDDC Manager alone.

The core order does not change based on your opinion. NSX Manager goes first, then vCenter, then the ESX hosts, then the NSX Edge cluster to finalize NSX. One change worth calling out for 9.1: the NSX host bits now ride along with the ESX upgrade, and the NSX Edge cluster is upgraded on its own as the last step. So the Edge cluster shows up at the end rather than next to NSX Manager.

NSX Managerstep 1vCenterstep 2ESX hostsstep 3, rollingNSX Edgestep 4, finalizevSAN, vDSoptionalNSX host bits ship with the ESX upgrade, so the Edge cluster finalizes NSX at the end
Figure 2. The fixed order for the management domain core in 9.1.

Which vCenter and NSX path should you pick?

When you plan the domain upgrade, 9.1 gives you two ways to handle vCenter and NSX Manager together. The optimized path prepares a new vCenter instance for migration, upgrades NSX, and then switches over to the new vCenter. The sequential path upgrades NSX Manager first and then upgrades vCenter in place. Optimized shortens the overall window because the new vCenter is staged while NSX is working, but it introduces a switchover you have to plan around. Sequential is slower on paper and easier to explain to a change board.

Table 2. Rough timings for a small management domain, four hosts. Figures are field estimates and vary with hardware and load.
StepOptimized, roughSequential, roughImpact
SDDC Manager40 to 60 min40 to 60 minMgmt plane only
NSX Manager60 to 90 min60 to 90 minControl plane, no data outage
vCenter30 to 45 min, staged60 to 90 min, in placeBrief mgmt gap at switchover
ESX, per host30 to 45 min30 to 45 minRolling, hosts drain via vMotion
NSX Edge cluster30 to 45 min30 to 45 minNorth-south path failover
02h4h6h8h10hOptimizedabout 7 hSequentialabout 8.5 hTotal window for a four-host management domain, rough field estimate
Figure 3. Optimized trims the window mostly by staging vCenter while NSX Manager upgrades. Real numbers depend on host count and hardware.

vSphere Lifecycle Manager images, not baselines

This one stops upgrades cold if you find it late. vSphere Lifecycle Manager baselines are not supported in 9.x. Before ESX can go to 9.1, any cluster still using baselines has to be transitioned to a vLCM image. If your management domain was built on 5.2 and carried baseline clusters forward, that transition is a real task you schedule ahead of the ESX step, not something the upgrade does silently for you.

Seen this go wrong: a domain planned fine and passed the top-level precheck, then the ESX step refused to start because one cluster was still on a baseline. The transition to an image took an evening on its own. Check every cluster for baselines during planning, not on the night of the ESX upgrade.

Upgrade ESX, then finalize with the Edge cluster

ESX upgrades roll host by host. Each host enters maintenance mode, its VMs vMotion off, the host updates and reboots, then it rejoins and the next host starts. On a four-host management domain that is four rounds of drain, patch, reboot, rejoin. Keep enough spare capacity that one host down does not starve the cluster, or hosts will sit waiting to enter maintenance mode.

Once every host is on 9.1, upgrade the NSX Edge cluster. That step finalizes the NSX upgrade and closes out the management domain core. After that you can pick up the optional items when you have a window: vSphere Distributed Switch versions, vSAN on-disk format, and vSAN file service. None of those are required to be at 9.1, but they turn on newer features and I usually do the vDS and vSAN format bumps within a week rather than leaving them hanging.

045m90m135m180mHost 1Host 2Host 3Host 4Hosts upgrade one at a time, roughly 45 minutes each, about 3 hours for four hosts
Figure 4. ESX is serial by design. Four hosts at about 45 minutes each land near three hours of rolling work.

Watch the upgrade and handle a paused step

VCF Operations shows the domain upgrade as a running task with per-component status, and that is the screen you keep open the whole window. It tells you which component is active, which are queued, and where a step has paused. Most stalls are not failures, they are the upgrade waiting on a condition it cannot satisfy on its own, and the status text usually names it.

The pauses I see most

The common one is a host that will not enter maintenance mode because there is nowhere for its VMs to go. If the cluster is tight on capacity, DRS cannot evacuate the host and the ESX step sits. Freeing capacity or relaxing a too-strict anti-affinity rule usually releases it. Another is an NSX Edge that fails its own precheck at the finalize step because an earlier host round left a transport node in a partial state. The upgrade log files, reachable from the same VCF Operations area, are where you confirm which node is unhappy before you touch anything.

When a step pauses, resist the urge to cancel and restart the whole plan. Read the status, fix the named condition, and retry the single step. A full restart re-runs work that already succeeded and lengthens the window. If a step genuinely fails rather than pauses, that is when the snapshot and backups you took earlier earn their keep, and Part 14 goes deep on rolling back a stalled upgrade.

A worked example: a four-host management domain

Take a small management domain with four ESX hosts, one NSX Manager cluster, one vCenter, and a single NSX Edge cluster. I chose the optimized vCenter and NSX path. Here is roughly how the evening went, and these numbers are the ones plotted in Figure 4.

SDDC Manager took about 50 minutes from VCF Operations. I then fixed the depot to the activation code, which cost 10 minutes I had forgotten to budget. NSX Manager ran about 80 minutes. vCenter was staged during the NSX work and the switchover added a short management gap of a few minutes. The four ESX hosts rolled at 40 to 45 minutes each, so about three hours of drain, patch, reboot, rejoin. The NSX Edge cluster finalized in about 35 minutes. Add the precheck reruns and coffee, and the whole management domain landed inside a single long maintenance window rather than spilling into a second night. The one lesson I carry forward: budget the depot re-auth and a baseline check as real line items, because those are the two steps that turn a clean plan into a stall.

What I would do: for anything past a lab, use the optimized vCenter path only if you have practiced the switchover once. If this is your first 9.1 management domain, take the sequential path. It is slower by an hour or so, and that hour buys you a simpler mental model when something needs a decision at 1 a.m.
Signs it worked: SDDC Manager reports 9.1 in VCF Operations, the domain upgrade plan shows every core component at the target, NSX Manager and the Edge cluster both read 9.1 with a healthy state, all four hosts show 9.1 and are out of maintenance mode, and vCenter answers on the expected address after the switchover.
Caution: the ESX and Edge steps touch running workloads and the north-south path. Do them in a change window, confirm your backups and the SDDC Manager snapshot are current, and know your rollback before you start. A management domain upgrade is not risk-free, and treating it as routine is how a short window becomes a long night.

Common questions

Do I have to upgrade workload domains at the same time?
No. Upgrading workload domains is not a mandatory part of reaching 9.1 and can run later as a Day-N job. The mandatory scope is the fleet level and the management domain. I cover workload domains in Part 12.

Why did my depot stop working right after SDDC Manager hit 9.1?
Because 9.1 changes depot authentication. The old download token is replaced by an activation code from the VCF Business Services console. Reconfigure Depot Settings before you plan the domain upgrade and the bundles pull normally.

Can I still use vSphere Lifecycle Manager baselines?
No. Baselines are gone in 9.x. Every cluster has to be on a vLCM image before ESX goes to 9.1, so check for baseline clusters during planning and transition them first.

Optimized or sequential for vCenter and NSX?
Optimized is faster because vCenter is staged during the NSX work, but it adds a switchover. Sequential upgrades NSX Manager then vCenter in place and is easier to reason about. First time out, I take sequential.

Where does the NSX Edge cluster fit?
Last. NSX host bits ship with the ESX upgrade, so after hosts are on 9.1 you upgrade the Edge cluster to finalize NSX and close out the domain core.

VCF 9.1 Upgrade · Part 11 of 14
« Previous: Part 10  |  Complete Guide  |  Next: Part 12

References

Related on this site: VCF 9.0 vs 9.1 and the VCF 9.1 Upgrade Complete Guide.

About The Author


Discover more from Journal of Intelligent Infrastructure – By Dr Pranay Jha

Subscribe to get the latest posts sent to your email.

Leave a Reply

Your email address will not be published. Required fields are marked *

Architect’s Toolkit

About the Author

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

Discover more from Journal of Intelligent Infrastructure - By Dr Pranay Jha

Subscribe now to keep reading and get access to the full archive.

Continue reading