TL;DR · Key Takeaways
- Fleet Management gives you one console across every site. Cloud proxies are the local collection points that feed it, so you collect near the data and analyze in one place.
- A single proxy per site is a single point of failure. In 9.1 you put two or more proxies in a collector group, and if one drops, another picks up the collection.
- Shard the load. One proxy collecting 6,000 objects becomes two proxies at about 3,000 each, which is both faster per cycle and survivable.
- On one site, a lone proxy died on a Friday evening and collection was blind for roughly 60 hours until Monday. A collector group would have failed over with no gap.
- Size the analytics tier to the object count and keep headroom. My planning figures run from a couple of nodes for a small fleet to several for tens of thousands of objects.
- Do not collect everything centrally over the WAN. Put proxies local, group them, and let the central cluster do the analytics.
One cluster is easy to operate. The trouble starts at three sites and twenty clusters, when the console still looks fine but the collection quietly stops being reliable. Scale in VCF Operations is less about the dashboard you look at and more about the plumbing behind it: where data is collected, how that collection survives a failure, and whether the analytics tier is sized for the load you actually run.
This part is about running VCF Operations across many clusters and several sites without the collection layer becoming the weakest link. I will cover cloud proxies and collector groups, how I shard the load, how I size the tier, and a real weekend where a single proxy failed and nobody noticed for two days.
One console, many collection points
Fleet Management is the top of the stack, the single interface that builds, operates and secures the private cloud across the whole estate. Underneath it, cloud proxies do the collecting. A cloud proxy sits near the objects it monitors, gathers their metrics and logs, and ships the results back to the central analytics cluster. The point of the design is locality. You collect close to the data, across whatever geography your datacenters span, and you analyze in one place. A unified cloud proxy even handles the log collection work alongside metrics, so one appliance covers both.
Worth saying plainly: the central analytics cluster is itself a thing you have to protect. If it is the single place all your operational visibility lives, then its own availability and backup matter as much as any workload’s. Treat it as a tier-one service, give it the same backup and recovery attention you give production, and know how you would bring it back if the site hosting it had a bad day. Losing the console does not stop the workloads, but it does blind you across every site at once, which is a bad state to be in during an incident.
Why one proxy per site is a trap
The obvious build, one proxy per site, is the one that bites you. A proxy is a running appliance, and appliances fail, reboot and get disconnected. When the only proxy at a site goes down, that whole site stops reporting, and because the console still shows the last data it had, the outage can hide in plain sight. In 9.1 the answer is a collector group. You put two or more cloud proxies in a group, and if the one currently collecting fails or disconnects, another proxy in the group takes over. Collection continues, the site keeps reporting, and you find out from an alert rather than from a gap you notice on Monday.
Grouping does two jobs at once. It gives you high availability, and it lets you balance load. Instead of one proxy carrying every object at a site, the group spreads the objects across its members, so each proxy does less and finishes its cycle faster. The chart below is a site that moved from one proxy carrying 6,000 objects to two proxies carrying about 3,000 each. Same objects, half the load per appliance, and now survivable.
| Aspect | Single proxy | Collector group |
|---|---|---|
| Failure of one proxy | Site goes blind | Another proxy takes over |
| Load per appliance | All objects on one | Spread across members |
| You find out from | A gap, eventually | An alert, at failover |
The weekend a proxy died quietly
Here is the anecdote that made me a believer in groups. A remote site ran a single proxy. It dropped its connection on a Friday around 6pm. The central console still displayed the site, just with data that stopped updating, and nobody was staring at that site over the weekend. Collection for that site was effectively blind from Friday evening until someone caught it Monday morning, roughly 60 hours. Nothing broke on the workloads, but for two and a half days we had no operational visibility into a production site and did not know it. A collector group would have failed over in minutes and raised an alert instead of a silence.
Sizing the tier for the load
Proxies collect, but the central analytics cluster has to store and process everything they send, and that scales with object and metric count. This is where people under-build and then wonder why the console is slow and collection cycles run long. I size the analytics tier to the total object count with headroom, and I add proxies per site to keep each one comfortably under load. The figures in the table are my planning numbers, not a vendor limit, and I treat them as a starting point to validate against the current sizing guidance for your release. The shape is what matters: nodes grow with objects, and you plan for the fleet you will have in a year, not the one you have today.
| Fleet size | Objects | Analytics nodes | Proxies per site |
|---|---|---|---|
| Single site | ~6,000 | 2 | 2 in a group |
| Three sites | ~18,000 | 3 | 2 in a group each |
| Large estate | ~40,000 | 5 | 2 to 3 in a group each |
Sharding the load on purpose
Grouping proxies gives you the option to spread load, but the spread is not always even by default, and an unbalanced group is only half the win. When you add proxies to a group, the aim is that each carries a similar slice of the objects and finishes its collection cycle with room to spare. I check the split after any change, because a group where one proxy quietly ends up with two thirds of the objects has the same overrun risk as a single proxy, just better disguised. Adapter instances are the unit you are really moving here, so think in terms of which adapter loads land on which proxy rather than counting appliances alone.
The temptation at scale is to keep adding objects to an existing group until it strains, because adding a proxy feels like more work than it is. Resist it. A proxy that is comfortable at 3,000 objects is not comfortable at 5,000, and the first sign of trouble is cycles that creep past the collection interval. When the split starts looking lopsided or the busiest member runs long, add a proxy and rebalance rather than hoping the current members absorb the growth. Rebalancing early is cheap. Doing it during an incident is not.
What to watch as you scale
Scale changes which numbers matter. On one cluster you watch workloads. Across a fleet you watch the collection layer itself, because that is what fails quietly. The first signal is collection completeness, the share of expected objects actually reporting, which should sit at 100 percent and which drops the moment a proxy or an adapter falls over. The second is cycle time against the collection interval. If a proxy takes longer to collect than the interval between collections, it is falling behind and your data is arriving stale, which is the late-alert failure from the anecdote above.
The third set is the health of the analytics tier itself: node CPU, heap, and whether the cluster is keeping up with ingest. An under-sized tier shows up as a slow console and delayed processing long before it shows up as an outage, so it is worth catching early. I put completeness, cycle time and node health on a small operations-of-operations dashboard, because the monitoring platform needs monitoring too, and at fleet scale it is the piece most likely to be neglected until it bites.
Keeping a global console useful per site
A single console across twenty clusters is only an asset if you can still ask about one site without drowning in the other nineteen. This is where the earlier parts pay off. A dynamic custom group per site, driven by a location property, gives you per-site rollups from the same super metrics you built once. Policies scoped per site let a busy production datacenter run tighter thresholds than a lab without forcing one setting on everyone. Filters on a shared dashboard let an operator point the same board at Site A now and Site B next. The fleet view and the per-site view are the same data, sliced by the groups and policies you set up, not separate tools you maintain twice.
If you are rolling this out as part of a version move, the collector group and sizing work belongs in the upgrade plan, not bolted on after. I set out that sequencing in the VCF 9.1 upgrade series, and the multi-site collection layer is one of the parts most worth rehearsing before you touch production.
Fleet-wide settings and tags in 9.1
VCF 9.1 pushed more of this to the fleet level, which helps when you run many sites from one console. You can manage vCenter tags across every vCenter in the fleet from a single pane, so a workload tagged for production in one site is not invisible when you look fleet-wide. Fleet Settings let you set DNS, NTP and password policies once and apply them across the fleet, and API access moved to OAuth 2.0 tokens with client management. For a multi-site operator that cuts the number of places a setting can drift. Wire your per-site checks to read these fleet settings, so a local override shows up as drift rather than as a surprise during an audit.
Common questions
How many proxies should a site have?
At least two, in a collector group, for any site you care about. That gives you failover and lets you spread the object load so no single appliance carries the whole site.
Can one proxy handle both metrics and logs?
Yes. A unified cloud proxy covers metric collection and the log collection work together, so you do not need a separate appliance for logs.
What happens when a collecting proxy fails?
In a collector group, another proxy in the group picks up the collection. The site keeps reporting and you get an alert at failover instead of discovering a gap later.
Should I collect everything centrally over the WAN?
No. Put proxies local to each site so collection is close to the data, and let the central analytics cluster do the processing. Hauling raw collection across the WAN is fragile and slow.
How do I size the analytics cluster?
To the total object count with headroom, and against the current sizing guidance for your release. Plan for next year’s fleet, not today’s, because retrofitting nodes under load is the hard way.
Does the central cluster need its own protection?
Yes. It is the one place all your cross-site visibility lives, so treat it as a tier-one service with real backup and a recovery plan. Losing it blinds every site at once, which is exactly when you least want to be blind.
How do I know a group is balanced?
Check the object split and the per-proxy cycle time after any change. If one member carries most of the objects or runs long, add a proxy and rebalance rather than letting the busiest one drift toward overrunning the interval.
« Previous: Part 14 | VCF 9 Operations Complete Guide | Next: Part 16 »
References
- VCF Operations: Collecting Data with Cloud Proxy
- VCF Blog: Scale, Simplify and Secure Private Cloud Operations with VCF 9.1
- VCF Operations: Cloud Proxy FAQ
About The Author
DrJha