This part rebuilds a lost SDDC Manager, vCenter Server or NSX Manager from the file based backups you configured in Parts 8 and 9. Every fleet should rehearse it, and run it for real when a management component fails. It depends on a reachable SFTP target, a matching appliance OVA for each version, and the encryption passphrase from your vault.
- Prerequisites: the encrypted backup on your SFTP target, a matching OVA for each appliance version, the encryption passphrase, and the same FQDN and IP as the lost appliance.
- Restore order when more than one component is lost: SDDC Manager first, then each vCenter Server, then NSX Manager.
- SDDC Manager restore deploys a fresh OVA, decrypts the backup, then runs the restore task through the VCF API.
- vCenter Server restore uses the vSphere GUI installer in two stages, stage 1 deploys the OVA and stage 2 copies the data.
- NSX Manager restore deploys one new node, then runs Start Restore inside NSX Manager under Backup and Restore.
- Revalidate SDDC Manager state after any vCenter restore, and confirm each component before you move to the next.
Restore is the other half of the backup work you did earlier in this series. In Part 8 you set the SDDC Manager and fleet backup, and in Part 9 you set the vCenter and NSX Manager backups. This part uses those copies to bring a failed component back. Each VCF 9.1 management appliance restores through its own path: SDDC Manager rebuilds from an OVA plus a restore task, vCenter Server rebuilds through the vSphere installer, and NSX Manager rebuilds from inside its own interface. You will walk all three, in the order that keeps the fleet consistent.
You need the encrypted backup files on the SFTP host, the encryption passphrase you stored in your vault, and the OVA image that matches each appliance version. Reserve the original FQDN and IP address of every failed appliance, because each restore expects to land on the same identity it was backed up from. Credential handling for these accounts lives in Part 6, and the operating model that explains why VCF Operations and SDDC Manager split these duties is in the series overview.
Plan the restore sequence
Decide the scope before you deploy anything. If only one component failed, restore that component alone and leave the rest of the fleet running. If you lost the management domain or several appliances at once, restore in a fixed order so each layer can find the one below it. Bring SDDC Manager back first, because it holds the inventory that ties the fleet together. Restore each vCenter Server next, so the domains and clusters reappear. Restore NSX Manager last, because it registers against a working vCenter.
Match every replacement appliance to the exact version it was backed up from. A backup taken on a given build restores only onto an appliance deployed from the same build, so confirm the version recorded in the backup listing and download the matching OVA before you start. Reserve the identity too. Keep the original FQDN, IP address, DNS records and NTP source for each appliance, because a restore that lands on a new address will fail its validation step.
Line up your inputs so the run is a lookup, not a scramble. Note the SFTP host, directory and service account from Part 9, the encryption passphrase from your vault, and the Single Sign On administrator credentials for vCenter. Where a neighbouring part owns a related step, use it rather than repeating it: backup configuration stays in Parts 8 and 9, and password recovery stays in Part 6. This part covers only the act of restoring.
Prerequisites
| Requirement | Detail | Where to confirm |
|---|---|---|
| Backup files on SFTP | The dated, encrypted archives written by the Part 9 jobs | SFTP server |
| Encryption passphrase | The same passphrase used at backup time, required to decrypt | Password vault |
| Matching OVA images | SDDC Manager, vCenter and NSX Manager OVA for the backed up build | Broadcom depot |
| Original identity | The same FQDN, IP address, DNS and NTP as each failed appliance | Your runbook |
| SSO credentials | The vSphere Single Sign On administrator account for vCenter recovery | Password vault |
| Jump host with OpenSSL | A workstation that can reach SFTP and decrypt the SDDC Manager archive | Admin workstation |
Restore in VCF 9.1 assumes the replacement appliance is identical in version and identity to the one it replaces. Gather the OVA, the passphrase and the original addresses first, and the console steps that follow become mechanical.
Step 1, Identify the failure and confirm the backups
Confirm which appliance is lost and that a usable backup exists before you deploy anything. Run these checks from a jump host that can reach the SFTP target.
- List the SFTP directory for the failed component and note the most recent dated archive.
- Confirm the archive size looks complete and its timestamp is recent enough for your recovery goal.
- Read the backup listing and record the version and the original FQDN and IP address it was taken from.
- Download the matching OVA for that exact version from the Broadcom depot.
- Retrieve the encryption passphrase for that appliance from your vault.
sftp svc-vcf-bck@192.168.10.20 sftp> cd /backups/sddc-manager sftp> ls -lt
A confirmed archive and a matching OVA are the two facts that make the rest of this procedure safe. Do not deploy a replacement appliance until both are in hand, because a restore onto the wrong build will stop partway and leave you deploying twice.
Step 2, Restore SDDC Manager from its file based backup
SDDC Manager restores by deploying a fresh appliance and then applying the decrypted backup through the VCF API. Use the SDDC Manager console path through vCenter for the OVA, then the appliance shell for the restore task.
- Download the latest SDDC Manager archive from SFTP to your jump host.
- Decrypt and extract the archive with the passphrase from your vault.
- In the vSphere Client, right click the management cluster and select Deploy OVF Template, then point to the SDDC Manager OVA.
- In the deployment wizard, set the same FQDN and IP address as the failed appliance, then click Finish.
- Power on the appliance and wait for its services to reach a running state.
- Copy the extracted backup to the new appliance, then open an SSH session as the vcf user and switch to root.
- Request an API token from the tokens endpoint, then start the restore against the restores endpoint.
- Poll the returned restore task until its status reports Successful.
# decrypt and extract on the jump host
OPENSSL_FIPS=1 openssl enc -d -aes-256-cbc -md sha256 \
-in sddc-manager-backup.tar.gz | tar -xz
# request an API token from the new SDDC Manager
curl -k -X POST https://sddc-manager.vcf.local/v1/tokens \
-H 'Content-Type: application/json' \
-d '{"username":"admin@local","password":"REPLACE_ME"}'
# start the restore task with the extracted backup and passphrase
curl -k -X POST https://sddc-manager.vcf.local/v1/restores/tasks \
-H 'Authorization: Bearer TOKEN' \
-H 'Content-Type: application/json' \
-d '{"elements":[{"resourceType":"SDDC_MANAGER"}],"backupFile":"/tmp/restore/sddc-manager-backup.json","encryption":{"passphrase":"REPLACE_ME"}}'Keep the appliance offline from the fleet until the restore task finishes, so it does not register in a half restored state. Field names in the restore payload can vary slightly by build, so confirm them against the Broadcom restore article for your exact version before you post the task.
Step 3, Restore a vCenter Server instance with the installer
vCenter Server restores through the vSphere GUI installer, not through SDDC Manager. Run the installer from a workstation that can reach the target ESXi host and the SFTP backup.
- Mount the vCenter Server installer ISO that matches the deployed version, then launch the installer.
- On the installer home screen, click Restore.
- On Enter backup details, type the SFTP location, user name and password, then click Next.
- Review the backup information shown, confirm the timestamp, then click Next.
- Complete stage 1 to deploy the new appliance onto the target host, keeping the same FQDN and IP address, then click Finish.
- When stage 1 completes, click Continue to move to stage 2.
- On the Single Sign On screen, type the SSO administrator user name and password, then click Validate and Recover.
- Wait for the data transfer to finish and the appliance to report a completed restore.
After a vCenter restore inside a VCF fleet, revalidate the SDDC Manager state so the inventory and the recovered vCenter agree. If SDDC Manager still shows the old appliance, follow the Broadcom instance recovery article to update the vCenter SSH keys held in the SDDC Manager backup, then recheck.
Step 4, Restore NSX Manager from inside NSX
NSX Manager restores from within NSX itself, and VCF discovers the recovered appliance afterward. Deploy one new node, point it at the same SFTP repository, and let NSX rebuild the cluster.
- In the vSphere Client, power off any surviving NSX Manager nodes so only the new node will run.
- Deploy one new NSX Manager node from the matching OVA, using the same FQDN and IP address recorded in the backup listing.
- Open a browser to https://<nsx_manager_fqdn> and log in as the admin user.
- Click the System tab.
- Under Lifecycle Management, select Backup & Restore.
- Next to the SFTP server, click Edit, confirm the repository details match Part 9, then click Save.
- In the list of available backups, select the timestamp you want, then click Restore.
- Follow the prompts, supply credentials if asked, and let NSX rebuild the cluster to three nodes.
Do not add the extra manager nodes yourself during the restore, because NSX recreates the cluster from the backup and adds nodes as part of that flow. Once the restore reports success, confirm the cluster shows three stable nodes before you hand the domain back to users.
How to verify the restore
Check each component before moving on. For SDDC Manager, SSH to the appliance and run the built in health check utility, then confirm every test reports green and the workload domains list correctly in the interface. For vCenter Server, log in to the vSphere Client and confirm the inventory, clusters and hosts appear as expected, then revalidate the SDDC Manager state so the fleet and the recovered vCenter agree. For NSX Manager, open System, Backup & Restore and confirm the restore shows success, then check that the cluster reports three stable nodes and that the transport nodes reconnect. A green SDDC Manager health check, a consistent vCenter inventory, and a stable NSX Manager cluster together confirm the restore is complete. Fleet wide health monitoring, which you use to watch for drift after a recovery, is covered later in this series.
Common errors and fixes
| Symptom | Cause | Fix |
|---|---|---|
| SDDC Manager restore task fails at the token step | Wrong admin credentials or services not fully started | Wait for services to reach running, then confirm the password using the guidance in Part 6 |
| vCenter stage 2 fails validation | SSO password mismatch, or FQDN and IP do not match the backup | Redeploy with the original FQDN and IP, and enter the correct SSO administrator credentials |
| NSX Manager cannot list any backups | SFTP repository or passphrase does not match the backup that was taken | Re enter the SFTP server details and passphrase from Part 9, then reload the backup list |
| SDDC Manager shows a stale vCenter after recovery | vCenter SSH keys changed during the vCenter restore | Update the vCenter SSH keys held in the SDDC Manager backup per the instance recovery article, then revalidate |
Common questions
What order should I restore VCF components in?
Restore SDDC Manager first, then each vCenter Server, then NSX Manager. That order lets each layer register against the one below it. If only one component failed, restore that one alone.
Do I restore vCenter from VCF or from the vSphere installer?
Use the vSphere GUI installer Restore option. VCF does not perform the vCenter file based restore for you, though you revalidate the SDDC Manager state afterward so the fleet agrees.
Can I restore NSX Manager from SDDC Manager?
No. NSX Manager restores from inside its own interface under System, Backup and Restore. VCF discovers the recovered appliance after NSX rebuilds the cluster.
What happens to workloads during a management component restore?
Running virtual machines keep running, because a management restore rebuilds the control plane rather than the workloads. Workload disaster recovery is a separate topic, covered later in this series.
References
- Broadcom TechDocs, Restore SDDC Manager from a File Based Backup
- Broadcom TechDocs, Restore a vCenter Instance from a File Based Backup
- Broadcom TechDocs, Validate the SDDC Manager State After a vCenter Instance Restore
- Broadcom TechDocs, Restore a Backup for NSX Manager
About The Author
