TL;DR · Key Takeaways
- Back up in the same order you upgrade: VCF Operations, then SDDC Manager, then NSX Manager, then vCenter. Every backup goes to an external SFTP target, not to a disk on the same appliance.
- vCenter and NSX Manager need file-based backups for real recovery. A VM snapshot rewinds one appliance, but it will not restore a distributed NSX database cleanly.
- There is no single button that rolls all of VCF back to 9.0.x. Rollback is per component, and for most of the stack it means restore-from-backup or redeploy, not undo.
- Take a vCenter snapshot right before its own upgrade, because the 9.1 flow uses a temporary IP and a switchover. Delete it the same day, once you have validated.
- Backup status is a precheck item in 9.1. If your last SDDC Manager or NSX backup failed, the precheck flags it, and that is a stop.
- When the upgrade validates, take fresh 9.1 backups of every component and remove the old snapshots so they do not rot on your datastore.
The upgrade steps get the attention, but the backup and rollback plan is the part that lets me sleep during the change window. I have never regretted a backup I took and did not need. I have very much regretted the one I skipped because the precheck was green and I was in a hurry. On a full-stack platform like VCF, the cost of being wrong is not a single VM, it is the control plane for everything.
So this part is about what to back up, how to back it up (file-based backup versus a snapshot, which are not the same thing), where each backup lands, and what rollback actually means for each component. I am writing from real 9.0.x to 9.1 upgrades, so the boundaries here are the ones I plan around, not a tidy promise from a datasheet.
Why there is no single rollback button
VCF is not one appliance. It is a set of dependent components, and each one keeps its own state in its own store: VCF Operations, SDDC Manager, NSX Manager, vCenter, and the ESX hosts all hold data that has to stay consistent with each other. There is no consolidated snapshot of the whole fleet you can revert to. That is the single fact that shapes the entire rollback plan.
What you get instead is a rollback boundary per component. If a step fails, you revert that one component to the state just before the failure, then troubleshoot from there. You do not roll the entire platform back to 9.0.x in one move. Broadcom is honest about this: there is no easy button that reverses a whole VCF upgrade. Backup SDDC Manager first, and if a later step fails you can return to the point before that failure so you can investigate without losing the rest of your progress.
What rollback really means here
For most of the stack, rollback is not an undo. It is restore-from-backup, and sometimes redeploy-then-restore. A vCenter that fails its upgrade can often be reverted from the pre-upgrade snapshot or its file-based backup because the 9.1 flow builds a new appliance and switches over. An NSX Manager cluster that ends up in a bad state is rebuilt from its file-based backup, not rewound. That difference is why the backup method matters as much as the fact that you took one.
Back up in upgrade order
The backup order should mirror the upgrade order. You upgrade VCF Operations first, then SDDC Manager, then NSX, vCenter, and ESX, so that is the order I confirm backups in too. The reason is simple: the component you are about to touch is the one whose backup has to be freshest. A three-week-old SDDC Manager backup is not much comfort when SDDC Manager is next in the queue.
Here is the backup that I confirm for each component before I let the upgrade start. Note the destination column: a backup sitting on the same appliance you are upgrading is not a backup, it is optimism.
| Component | Backup method | Destination | When |
|---|---|---|---|
| VCF Operations | Appliance backup / supported snapshot of the cluster nodes | External store | Before the PAK update |
| SDDC Manager | File-based backup (plus snapshot if your change plan calls for it) | External SFTP | Before SDDC Manager upgrade |
| NSX Manager | File-based backup (not a snapshot as your recovery plan) | External SFTP | Before NSX upgrade |
| vCenter | File-based backup, plus a snapshot right before its upgrade | External SFTP + local snapshot | Before vCenter upgrade |
| Workload VMs | Your normal backup product (unchanged by the upgrade) | Backup infrastructure | Per existing schedule |
VCF Operations and the cloud proxy
VCF Operations goes first, so its backup is the one I confirm first. Take a supported backup or snapshot of the cluster nodes before the PAK update, and remember that the 9.1 upgrade migrates fleet management into the new services cluster and then decommissions the old fleet management appliance. Once that migration completes, rewinding VCF Operations gets awkward, so the useful revert point is the state just before the PAK is applied. If a cloud proxy sits in the first VCF instance, note its version too, because it upgrades alongside VCF Operations and you want a known-good record of where it started. None of this is exotic, but it is the part people skip because the real drama comes later in the sequence.
SDDC Manager
SDDC Manager takes a file-based backup to an external SFTP server. Confirm the last scheduled backup ran and completed, and that the SFTP target is reachable with the current credentials. If your change plan also allows an appliance snapshot of SDDC Manager, take one right before the upgrade as a fast local rewind. The file-based backup is your real recovery path; the snapshot is a convenience for the appliance itself.
NSX Manager
NSX is the component I am strictest about. NSX Manager is a three-node cluster with a distributed database, and recovery without a valid file-based backup is genuinely hard. Configure NSX backups to an SFTP server, confirm a recent successful backup, and confirm the backup passphrase is recorded somewhere you can reach at 2 AM. Do not treat a VM snapshot of the manager nodes as your NSX recovery plan; snapshotting a distributed database rarely restores into a healthy cluster.
vCenter
vCenter gets both. Take a native file-based backup for a clean, supported restore, and take a VM snapshot right before the upgrade starts. The snapshot matters here specifically because the 9.1 vCenter flow builds a new appliance, uses a temporary IP address during the process, and then switches over. If the switchover goes wrong, a snapshot from a minute earlier is the fastest way back. Delete that snapshot the same day, once you have validated the upgraded vCenter, because a lingering snapshot on vCenter causes storage growth and performance problems.
File-based backup or snapshot?
These two get treated as interchangeable, and they are not. A file-based backup exports a component config and state into a consistent archive that the product knows how to restore, on the same version. A VM snapshot freezes an appliance disk at a moment in time. For a single appliance that you are about to change and might need to rewind, a short-lived snapshot is a fine safety net. For a distributed system like NSX, a snapshot of one node in a three-node cluster does not give you a coherent restore point.
Rollback boundaries, component by component
Since there is no whole-fleet undo, you plan the rollback for each component separately. Here is how I think about the boundary for each one, and the gotcha that catches people.
| Component | Rollback method | Gotcha |
|---|---|---|
| VCF Operations | Restore appliance backup or revert snapshot before the PAK update | Fleet management migrates into the new services cluster; plan the revert before that transfer completes |
| SDDC Manager | Restore file-based backup from SFTP, or revert the pre-upgrade snapshot | Restore has to match the version you backed up; a half-upgraded state is not a restore point |
| NSX Manager | Rebuild cluster and restore file-based backup from SFTP | No clean snapshot rollback for the cluster; the backup passphrase is mandatory |
| vCenter | Revert pre-upgrade snapshot, or restore file-based backup | Snapshot must predate the switchover; the flow uses a temporary IP mid-upgrade |
| ESX host | Exit maintenance mode, retry, or reimage from the prior image | vLCM baselines are gone in 9.x; you are on images, so keep the prior image available |
A worked example
On a recent 9.0.2 to 9.1 job, the upgrade prerequisite check came back with one red line: the SDDC Manager backup had not run successfully in four days. The scheduled job was pointing at an SFTP account whose password had rotated the week before, so every nightly backup had been failing silently. Nobody noticed because nobody was reading the backup emails. Ten minutes to update the credential, one manual backup to confirm the SFTP target accepted it, and the precheck went green.
Later in the same window, the NSX upgrade stalled on an edge node that would not enter the upgrade cleanly. Because I had a confirmed NSX file-based backup on SFTP and the passphrase written down, the decision was calm instead of frantic: hold the NSX upgrade at that boundary, open a case with the backup in hand as the fallback, and finish the edge remediation the next morning rather than gambling on a half-finished cluster overnight. The backup did not get restored in the end, but knowing it was good is exactly what let me stop instead of pushing forward into trouble.
Common questions
Can I roll the whole VCF fleet back to 9.0.x if the upgrade goes bad?
No. There is no single fleet-wide undo. You revert the specific component that failed to its pre-upgrade state and troubleshoot from there. So the per-component backups, and the order you take them in, matter a lot.
Are VM snapshots enough on their own?
Not for the cluster components. A snapshot is a fine same-day rewind for a single appliance like vCenter, but NSX Manager is a distributed database and snapshots do not restore it cleanly. Use file-based backups for SDDC Manager, NSX, and vCenter, and treat snapshots as a short-lived extra.
Why does vCenter want both a backup and a snapshot?
The 9.1 vCenter upgrade builds a new appliance, runs on a temporary IP, and switches over. The file-based backup is your supported restore, and the snapshot gives you a fast rewind if the switchover misbehaves. Take the snapshot right before the upgrade and delete it the same day.
How fresh does a backup need to be before the upgrade?
Fresh enough that the precheck passes and that you would accept losing the changes since it ran. In practice I take a manual backup of each component right before its upgrade, on top of the scheduled ones, so the restore point is minutes old rather than hours.
References
- How to Upgrade to VMware Cloud Foundation 9.1 (VCF Blog)
- Upgrade, Backup and Restore (Broadcom TechDocs, VCF 9.1)
- Upgrading VCF 9.0.2 to 9.1: Practical Notes (Angry Admin)
About The Author
DrJha