Our first disaster recovery drill on VKS restored the whole application in 12 minutes, green pods across the board, and I nearly called it a win. Then the PostgreSQL primary came up asking for crash recovery and the data it held was 6 hours stale, because the nightly snapshot underneath it was crash-consistent, not application-consistent.

Where the migration stands and what breaks without a safety net

Last part made cluster upgrades and node scaling routine on VKS, and it leaned on one promise: a Velero backup exists before every one-way change. This part builds that promise into a standing safety net, the observability that tells you something is wrong and the backup and restore path that gets you back. On TKGI this layer came half assembled, BOSH watched the VMs, Ops Manager surfaced tile health, and backup was a bolt-on you rarely tested. On VKS you assemble it deliberately, and that is an upgrade in honesty even where it is more work.

Our running estate is the same one carried since Part 5: three clusters named dev, staging and prod, a payments application with a web tier and a PostgreSQL database on persistent volumes, and a couple of stateless services beside it. Production cut over two parts ago and TKGI is idle. What is missing is a monitoring stack that speaks Kubernetes rather than BOSH, and a backup regime proven by a real restore rather than a green backup status. Both gaps are quiet until the night they are not.

Here is the distinction that governs everything below. Backing up a cluster and being able to restore it are different claims, and only one of them shows up on a dashboard. A backup job that reports Completed can still be unrestorable, a missing snapshot class label, a storage class that does not exist on the target, a database captured mid-write. Treat every backup as unproven until a drill has restored it into a running cluster, and you close the gap between what your status page says and what your on-call actually has at 3 in the morning.

Preflight for a trustworthy backup

Prove four things before you trust a single backup on VKS: the Velero server and client versions match, the node-agent daemonset is running on every worker because CSI data movement will not run without it, a VolumeSnapshotClass carries the label Velero looks for, and your backup storage location reports Available. Skip the label check and your volume backups silently fall back to skipping the data, which you discover only when a restore comes up with empty disks.

Now the check that catches most first-time failures. Velero moves CSI volume data only when a VolumeSnapshotClass carries the label velero.io/csi-volumesnapshot-class set to true. The vSphere CSI driver ships a snapshot class, but nothing labels it for you, so the obvious assumption, that installing Velero with the CSI feature is enough, is wrong in practice. Label it once and every future volume backup picks it up.

Standing up observability on VKS

Monitoring on VKS follows three pillars, and the VCF reference architecture names a tool for each. Metrics run on the kube-prometheus-stack, a bundle of the Prometheus Operator for target discovery, Grafana for dashboards, and Node Exporter for node statistics. Logs flow through Fluent Bit, a lightweight log shipper, into Grafana Loki for storage and search. Traces use OpenTelemetry to emit spans, Jaeger to visualize them. Each pillar also forwards to VCF Operations, so a spike in a Grafana panel links back to the datastore or host underneath it.

Install the metrics pillar first, it is the one an on-call rota needs on night one. Add the Prometheus community chart, then confirm the operator, Grafana and the node exporters all reach Running. Give Prometheus a persistent volume and a retention window up front, because the default emptyDir loses your history on the first pod reschedule, which on a rolling cluster is often.

A failure I hit on the first install, and the fix. Prometheus stayed Pending for ten minutes with its claim unbound, because the cluster had no default storage class and the chart quietly requested one that did not exist. Naming the storage class explicitly, as above, cleared it. Read the pod event before you touch anything, the event line names the exact class the claim wanted.

Scheduled Velero backups for stateful and stateless tiers

Backups split cleanly by tier, and the two tiers want different schedules. Stateless services carry no data worth an RPO argument, so a daily object backup that captures Deployments, Services and config is plenty. The PostgreSQL tier is the one that keeps you honest, its data changes every second and a crash-consistent snapshot is not the same as a restorable database. Schedule both, but treat them as different problems.

Step one, schedule the stateless namespaces with a plain daily backup and a 30 day retention, expressed as a 720 hour time to live. A Velero schedule is a cron expression, so 0 2 * * * runs at 2 in the morning, and each run is saved as the schedule name plus a timestamp.

Step two, the stateful tier, where the volume data and the quiesce both matter. Turn on CSI snapshot data movement so the PostgreSQL volume is copied to backup storage, not just snapshotted in place, and add a pre-backup hook that runs a checkpoint so the on-disk state is consistent when the snapshot fires. Annotate the database pod, and read the password from the PGPASSWORD environment variable already set on the container, never a literal in the manifest.

Step three, verify the first stateful backup actually moved the volume, and watch for the failure that trips most teams. If the snapshot class was never labelled, the backup finishes PartiallyFailed and the persistent volume is skipped, so the backup looks done but is hollow.

Keep the matrix below within reach, it is the reference artifact for this part. It maps every part of the estate to the mechanism that actually protects it and the recovery point you get, so nobody assumes Velero covers a thing it does not. Notice the last row, the PostgreSQL data has two entries, because the honest answer is that Velero gives you a floor and application archiving gives you the real number.

Disaster recovery drill and restore verification

A backup you have never restored is a guess. Run the drill deliberately: stand up a fresh VKS cluster or a clean namespace, restore the latest backup into it, and verify the app serves traffic and the database holds real rows. The flow below is the one to internalize, because a controller-driven platform rebuilds a cluster from spec and then Velero rehydrates the state on top.

flowchart TD
  A[Latest Velero backup in object storage] --> B[Provision a fresh VKS cluster from spec]
  B --> C[velero restore create from backup]
  C --> D[Data mover rehydrates the PostgreSQL volume]
  D --> E{App serves and rows present}
  E -->|yes| F[Cut traffic to restored cluster]
  E -->|no| G[Read restore logs, fix storage class or hook, retry]
  G --> C

Now the restore failure that comes straight out of a migration, a storage class name that existed on TKGI but not on VKS. Your old volumes referenced a class like tkgi-fast, the target cluster names its class vsphere-csi-fast, and the persistent volume claim restores stuck Pending because the class it asks for is gone. Velero solves this with a change storage class mapping, a labelled ConfigMap it reads during restore.

Rollback and fallback here are gentle by design, because restore is additive. A restore into a fresh cluster never touches the running one, so a bad drill costs you a cluster you throw away, not an outage. Keep the original cluster serving until the restored copy passes its checks, then shift traffic. If a restore comes up wrong, read the restore logs, fix the storage class map or the missing hook, and run it again, the source backup is untouched and reusable.

The measurements below are from three drills on the running example, and they are the argument for archiving beside Velero rather than trusting it alone. A nightly-only backup leaves a 24 hour RPO. Tightening the Velero schedule to every 6 hours helps but caps at 6 hours and adds snapshot load. Continuous WAL archiving, the PostgreSQL write ahead log streamed to object storage, pulls the real recovery point under 15 minutes, which no snapshot cadence reaches without hammering the array.

Failure lookup for backup and recovery

Below is the lookup I keep open during recovery work. Every row is a symptom I have hit on this estate, its cause, and the fix, so a stalled restore is a page turn rather than a debugging session at the worst possible hour.

Backup and recovery verdict and Monday move

My worst recovery lesson on VKS is the one from the top of this post, dressed in numbers. My first drill restored the payments stack in 12 minutes and I logged it as a pass, because every pod was green. Two weeks later a real incident forced the same restore, and the PostgreSQL came up needing crash recovery and lost 6 hours of orders, because the nightly snapshot was crash-consistent and I had trusted Velero alone. I reversed that decision the same week: a pre-backup CHECKPOINT hook so the snapshot is clean, and continuous WAL archiving beside Velero so the recovery point is minutes rather than hours. That one change moved us from a 6 hour data loss to under 15 minutes, and it cost an afternoon.

Here is the verdict for anyone standing this layer up. For observability, run the kube-prometheus-stack in cluster and wire it to VCF Operations, and avoid the instinct to recreate your TKGI dashboards, because the BOSH signals they watched are gone and the Kubernetes ones you need ship in the community dashboards already. For data protection, pick Velero with CSI data movement and quiesce hooks as your floor, and put application-level archiving on top for anything transactional, and avoid the tutorial default of one nightly snapshot with no hook, because it restores a database that needs crash recovery and quietly loses your last hours of writes.

Two questions land on every recovery review. Is Velero enough for the database on its own? No, it is your floor and your portability, use it for the volume and objects, and pair it with application archiving for the recovery point a transactional system needs. Do we still need a monthly drill once restores are working? Yes, more than ever, because the failure that bites is drift, a label that fell off or a storage class that got renamed, and only a scheduled restore surfaces it before an outage does.

For the backup toolchain this stands on, the Velero toolchain part set up source and target, and the stateful migration part covered the PostgreSQL volume move this recovery protects. For the layers below, the VKS Complete Guide and the VCF 9 Complete Guide own the platform detail, and the TKGI to VKS guide holds the full map.

On Monday, run one restore drill you have never run. Pick the payments backup, restore it into a scratch namespace on a non-production cluster, and check that the database answers a real query with real rows. If it comes up needing crash recovery, you just found your quiesce gap in a drill instead of an incident, which is the entire point. Next part lands the migration, decommissioning TKGI and Ops Manager, and points at what to learn after the estate is fully on VKS.

References

VMware Cloud Foundation Blog, Observability on vSphere Kubernetes Service
Velero Docs, CSI Snapshot Data Movement
Velero Docs, Disaster Recovery

About The Author

Dr. Pranay Jha

Dr. Pranay Jha is a Cloud and AI Consultant with 18+ years of experience in hybrid cloud, virtualization, and enterprise infrastructure transformation. He specializes in VMware technologies, multi-cloud strategy, and Generative AI solutions. He holds a PhD in Computer Applications with research focused on Cloud and AI, has published multiple research papers, and has been a VMware vExpert since 2016 and a VMUG Community Leader.

See author's posts