Forty minutes after the 4.20.4 update landed, a capacity planner asked me for the p99 latency graph covering the previous fortnight, and I could not produce it. Prometheus was healthy, every Console dashboard rendered, and every one of them started at midnight that morning. Fourteen days of history had been sitting on an emptyDir volume, and the Machine Config Operator had rebooted the node underneath it.

Three Recovery Domains, Three Different Tools

On TKGI this question had a lazy answer. BOSH, the release engineering and VM lifecycle engine underneath TKGI, could recreate any VM it managed from its own database, and most teams backed up the BOSH director and the Tanzu Operations Manager installation, called it disaster recovery, and moved on. That answer worked because BOSH owned the machines. OpenShift does not work that way, and neither does the recovery story.

Break it into three domains and the tooling stops being confusing. Domain one is application state, meaning namespaces, Deployments, Secrets, ConfigMaps and the data on persistent volumes. That belongs to OADP (OpenShift API for Data Protection, the Operator that packages Velero), which we installed back in Part 17 to move workloads off TKGI and which now becomes a permanent operational service rather than a migration tool. Domain two is cluster state, meaning everything in etcd, the key value store that holds every resource object the API server knows about. That belongs to etcd snapshots and nothing else. Domain three is telemetry, meaning metrics, alerts and logs, which no backup product covers because losing it does not stop the platform, it only makes you blind.

Confusing the domains is how estates end up unprotected while believing they are covered. A weekly etcd snapshot does not contain one byte of your PostgreSQL data. An OADP backup of every namespace on the cluster does not restore a MachineConfig, an OAuth configuration, or a deleted ClusterRoleBinding. Both statements surprise people who ran TKGI, because on TKGI the platform and the workload were backed up by the same tile.

Prerequisites and Preflight

Everything below assumes the estate we have been building since Part 12, now carrying production traffic after the cutover in Part 23 and running the release we updated to in Part 24. You need cluster-admin, a working default StorageClass backed by vSphere CSI (Container Storage Interface, the plugin that provisions vSphere volumes), and the same S3 compatible bucket OADP has been writing to since Part 17. Run the preflight before anything else, because two of these checks fail on a default install and both failures are silent.

Check one is the finding that matters. An OpenShift cluster that has never been touched runs Prometheus and Alertmanager on emptyDir, which is a directory on the node that disappears with the pod. Red Hat documents persistent storage as highly recommended for production and as required for high availability on multi node clusters, but nothing in the installer or the Console warns you, and the stack looks perfectly healthy right up to the moment a node drains.

Steps 1 to 4, Storage and Retention for Platform Monitoring

Core platform monitoring is configured through a single ConfigMap named cluster-monitoring-config in the openshift-monitoring namespace. On a fresh cluster that ConfigMap does not exist, so step 1 is creating it. Steps 2 and 3 attach volume claim templates to Prometheus and Alertmanager and set retention explicitly rather than relying on the 15 day default. Step 4 sends a copy of the metrics somewhere the cluster cannot destroy.

Two details in that manifest are deliberate. Raw block volumes are rejected outright because Prometheus cannot use them, so volumeMode must be Filesystem. And retentionSize is set to 34GB against a 40Gi claim rather than something closer to the ceiling, because compaction only runs every two hours and a volume can overshoot its retention size in between. Leave less headroom than that and the KubePersistentVolumeFillingUp alert starts firing at three in the morning about a volume that is behaving exactly as designed.

Here is the failure I hit on the first attempt, on a cluster where the vSphere CSI StorageClass had been created without the default annotation. Prometheus never came back, and the reason was two objects away from the pod that was complaining.

Step 4, Getting Logs Off the Cluster

Logging is a separate Operator on OpenShift, not part of the core stack. Install the Red Hat OpenShift Logging Operator and the Loki Operator, create a LokiStack backed by the same object storage bucket family OADP already uses, then create a ClusterLogForwarder using apiVersion observability.openshift.io/v1. One change from older releases catches every team migrating from a Fluentd based TKGI setup: the collector service account must be granted log collection permission explicitly, through the collect-application-logs, collect-infrastructure-logs and collect-audit-logs cluster roles. Skip that and the ClusterLogForwarder reports Ready while forwarding nothing at all.

If you already run a central log platform, and most estates leaving TKGI do, point the ClusterLogForwarder at it and skip LokiStack entirely. Running Loki in cluster only to ship the same lines onward twice is storage you pay for and an Operator you have to upgrade.

Steps 5 to 8, etcd Backups You Can Actually Restore From

etcd is where the cluster keeps itself. A snapshot of it plus the static pod resources is the only thing that will bring a cluster back after quorum is lost or after someone deletes something structural. Red Hat ships a wrapper script, cluster-backup.sh, maintained as part of the etcd Cluster Operator, and it produces two files: a snapshot database and a tarball of static pod resources that also carries the encryption keys when etcd encryption is on.

Step 6 is copying both files off the node, together, to storage that survives the cluster. Step 7 is the rule that catches everybody. A snapshot only restores a cluster running the same z-stream release that produced it. A 4.19.9 snapshot cannot restore a 4.20.4 cluster, which means every etcd backup you hold becomes scrap the moment an update completes. Take a fresh snapshot immediately before you start an update and another one immediately after it finishes, and write the release into the filename so nobody has to guess later. Step 8 is verifying the copy landed intact, because a truncated snapshot fails at restore time and not before.

Steps 9 to 12, Scheduled Application Backups with OADP

OADP arrived in this series as a migration tool. From here it is a backup product, and the change is mostly one object: a Schedule, which is a Velero custom resource carrying a cron expression and a backup template. Step 9 splits the estate into two schedules, because stateless namespaces and stateful ones deserve different frequencies and very different runtimes. Step 10 applies them. Step 11 sets a retention window with ttl. Step 12 restores something on purpose, into a scratch namespace, to prove the backups are real.

That failure is the same Security Context Constraint problem this series has been warning about since Part 7, and it is worth seeing here rather than during a real incident. SCC is OpenShift admission control, and restricted-v2 is the default posture. A restore into a new namespace gets a new allocated UID range, so a manifest with a hardcoded runAsUser that was admitted in shop-db is rejected in shop-db-drill. Either drop runAsUser from the pod spec and let the namespace annotation supply it, or bind the workload service account to nonroot-v2. Details are in Part 19. The point for this Part is narrower: a restore is a fresh admission decision, so a backup that restores cleanly into its original namespace can still fail into a different one.

flowchart TD
  A[Incident declared] --> B{What is missing}
  B -->|One workload revision| C[oc rollout undo]
  B -->|Namespace or its data| D[OADP restore from schedule]
  B -->|Cluster scoped objects| E[etcd restore to previous state]
  B -->|One control plane node| F[Replace unhealthy etcd member]
  D --> G[Recheck SCC admission on restored pods]
  E --> H[Full API outage, plan hours not minutes]
  F --> I[No API outage, quorum preserved]
  C --> J[Service restored]
  G --> J
  H --> J
  I --> J

Verification, Rollback and Clean End State

Verification for backup work is unusual, because a green status field proves almost nothing. A Backup in phase Completed means Velero finished writing to a bucket. It does not mean the contents will start. Everything below is either a check that the plumbing exists or a check that a restore actually produced a running workload.

Rollback here is refreshingly cheap, which is the one advantage of doing this work while TKGI is still powered on. Removing the volumeClaimTemplate stanzas from cluster-monitoring-config and reapplying puts the stack back on emptyDir within four minutes, at the cost of the history on those volumes. Deleting a Schedule stops future backups and leaves every existing Backup object and its bucket contents untouched. Neither action can take an application down. Only one operation in this Part is genuinely dangerous, and that is restoring etcd to a previous cluster state, which Red Hat describes as destructive and destabilising and a last resort. Treat it as the thing you do after the other three options have been ruled out, not the first lever you pull.

Common Failures and Remediation

Do I still need Velero on the TKGI side? Only until the last workload has moved and the last restore drill has passed. After that, retire it with the cluster in Part 26. Can OADP back up the OpenShift cluster itself? No. It backs up namespaced resources and volume data. Cluster scoped configuration belongs to etcd snapshots and to whatever Git repository holds your MachineConfigs. Does a bigger Prometheus volume buy longer history? Only alongside a longer retention setting. Retention time and retention size are both ceilings, and whichever is hit first wins.

Default Recovery Posture for a Migrated Estate

So here is the posture I now set on day one of any migrated cluster, before the first application arrives. Persistent volumes on Prometheus and Alertmanager with retention stated explicitly, because the default is amnesia. Remote write to something outside the cluster for anything you will need during an incident that takes the cluster down. Two OADP Schedules, split by whether volumes are involved, with a ttl that matches what compliance actually asked for. An etcd snapshot taken by an external scheduler, never by the Technology Preview feature, with the release baked into the filename and a fresh pair taken either side of every update. And one restore drill a month into a scratch namespace, on the calendar, owned by a name.

Teams landing on VMware vSphere Kubernetes Service instead of OpenShift face the same three domains with different plumbing, covered in the TKGI to VKS guide. Either way the lesson holds: on TKGI one tile protected the platform and its workloads together, and no platform you migrate to will do that again.

Do this on Monday. Run oc -n openshift-monitoring get pvc against your own cluster. If it says no resources found, you are one node drain away from the same phone call I took, and you can fix it in a single apply before lunch.

References

• Red Hat, Backing up and restoring etcd data, OpenShift Container Platform 4.19
• Red Hat, Storing and recording data for core platform monitoring
• Red Hat, OADP application backup and restore, OpenShift Container Platform 4.19
• Red Hat, Configuring log forwarding, Red Hat OpenShift Logging 6.3

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts