Your GitOps Hub Will Become the Bottleneck Long Before Cluster Count Tells You

GitOps hub bottlenecks are usually predicted more accurately by watched object volume, reconcile queue depth, and controller memory growth than by cluster count alone. Large-scale testing described in the discussion showed Argo CD application controllers hitting out-of-memory failures around 15,000 to 20,000 cached objects per hub, while sharding and tuning delayed the limit without removing the underlying memory cost.

The most important lesson was uncomfortable because it challenged the usual instinct to keep tuning the existing platform. At very large scale, architecture mattered more than configuration. Hydrated manifests helped. More replicas helped. Dynamic sharding helped. None of them changed the fact that a centralized reconciliation model still had to hold and process a huge amount of state.

The testing was not presented as a universal benchmark or as proof that one tool always beats another. It was a record of one setup, built through dozens of iterations over several months, and the failures were as valuable as the successful runs. That is exactly why the results matter. They show where teams should look before the hub becomes the thing taking the fleet down.

Cluster count is the wrong first metric

A fleet with 1,000 tiny clusters may place less pressure on a GitOps control plane than a much smaller fleet containing thousands of applications and deeply expanded resource trees. The number of managed clusters is visible and easy to report, but it does not describe the controller’s actual workload.

The more useful mental model is objects over clusters.

Each application contributes desired state, live state, cached trees, reconciliation work, and queue activity. A cluster that runs a few small addons may be cheap to manage. Another cluster with many applications and large manifest sets may consume far more memory and reconciliation time. That means two fleets with the same cluster count can behave very differently.

One commenter described watched objects and application controller queue depth as the indicators that predicted the wall in their environment. That matched the reported failure range of roughly 15,000 to 20,000 cached objects per hub. The controller was not failing because it had counted too many clusters. It was failing because one process was trying to watch and hold too much state.

This distinction changes capacity planning. Teams should stop asking only how many clusters a hub can manage and start asking how many objects, applications, watches, and queued reconciliations each controller process can sustain.

OOM kills are an architectural warning

An out-of-memory kill can look like a resource sizing problem. Sometimes it is. At fleet scale, repeated application controller OOMs can also be a sign that the control plane is carrying a workload its reconciliation model handles inefficiently.

The discussion described Argo CD consuming about 21 GB of memory in one addon-style rollout scenario, while Sveltos handled a similar pattern with roughly 2 GB. The author was careful not to frame that as a general benchmark claim. The comparison belonged to a specific setup and use case. Still, the size of the gap was difficult to ignore.

The reason was not simply better tuning. The tools approached the problem differently.

A centralized controller that caches and reconciles large resource trees will accumulate state as the fleet grows. A model that pushes lightweight agents into managed clusters can distribute classification, drift detection, and reconciliation work closer to the spokes. The memory profile changes because the architecture changes.

That is the point teams often resist. They keep increasing limits, adding replicas, and tuning flags because replacing or combining control-plane components feels more disruptive. But when memory grows with the amount of watched state, tuning may buy time without changing the curve.

A hub that repeatedly recovers from OOM kills is not healthy because it comes back. It is warning that the design is approaching a boundary.

Why sharding helps and still disappoints

Sharding the Argo CD application controller was one of the most obvious responses, and the testing did include it. The setup used between three and six controller replicas and tested dynamic cluster distribution.

It helped. Load was spread more evenly. No single controller had to hold the entire fleet’s cache. The failure arrived later.

But the total memory requirement remained.

That led to one of the clearest conclusions in the discussion: sharding spreads memory usage, but it does not reduce it. If the fleet requires a certain amount of cached state, dividing that state across more processes can reduce the pressure on each process, but the aggregate cost still exists. The operational burden may also grow because teams now manage more replicas, more shards, more balancing behavior, and more failure modes.

Sharding remains valuable. It can create runway, isolate failures, and improve controller distribution. It is especially useful when the current architecture is still fundamentally sound and the goal is to avoid one overloaded process.

The mistake is treating sharding as proof that the scalability problem is solved. It may postpone the wall while leaving the same relationship between fleet size and control-plane memory intact.

Capacity planning should therefore measure both per-shard pressure and total control-plane cost. A system that scales only by continually adding memory and controller replicas may still be moving toward an expensive and fragile design.

Hydrated manifests and tuning can only go so far

The testing also found that hydrated manifests improved behavior. Pre-rendering or otherwise reducing work required during reconciliation can remove repeated computation and reduce some of the load placed on controllers.

That kind of optimization matters. Large fleets amplify small inefficiencies, so reducing manifest generation work, avoiding unnecessary reconciliation, tightening selectors, and improving distribution can produce meaningful gains.

Yet the discussion repeatedly returned to the same limitation: tuning helped only partially.

This is a familiar pattern in infrastructure engineering. Teams optimize the obvious hot path, then discover that the real cost is structural. A controller still has to know about the desired state. It still has to compare it with live state. It still has to hold or retrieve enough information to make reconciliation decisions. The exact memory profile may improve, but the relationship between managed state and controller pressure remains.

Tuning should be treated as a way to validate the architecture, not as a substitute for architecture.

A sensible sequence is to reduce unnecessary work first, then observe whether the resource curve becomes manageable. If memory still grows too quickly with watched objects, the next question should not be which flag has not been tried. It should be whether the reconciliation model matches the scale and workload pattern.

One controller per cluster changes the tradeoff

Several people in the discussion described a different design: place Argo CD or Flux CD in each managed cluster, then use central configuration, generators, or automation to deploy the correct application sets to the correct locations.

This approach distributes reconciliation by default. Each cluster handles its own local state, so one central application controller does not need to cache the entire fleet.

The tradeoff is operational duplication.

Running a GitOps controller in every cluster means every location needs enough CPU and memory for that controller. Upgrades, configuration, credentials, and lifecycle management must be handled across the fleet. For well-resourced clusters, this may be acceptable. For small edge locations, even Argo CD Core may be too heavy.

That edge constraint was central to the original setup. The fleet involved resource-limited locations, which made a fully decentralized Argo CD deployment unattractive. A centralized management model was chosen partly because it reduced spoke-side overhead.

This is why there is no universal winner.

A per-cluster model may scale more naturally and provide a stronger security posture because spokes can pull from central configuration without allowing the hub broad inbound access. A centralized model may simplify governance and reduce component duplication. An agent-based model can offer a middle path by placing lightweight components in the spokes while keeping a central view.

The right design depends on where the organization can afford complexity: in the hub, in every cluster, or in the coordination layer between them.

ApplicationSets solve targeting, not infinite scale

ApplicationSets were discussed as an important fleet-management mechanism. Their generators can dynamically create Argo CD Applications based on lists, cluster secrets, Git data, matrix combinations, merges, and other inputs.

In the described setup, a cluster generator used label selectors on registered cluster secrets. A label such as cert-manager: enabled could cause the generator to create an application for every matching cluster and deploy the relevant addon to one, ten, or one hundred locations.

This is operationally useful because it creates manageable fleet slices. Labels define which clusters receive which applications. Progressive syncs then allow teams to roll changes out in controlled waves rather than updating the entire fleet at once.

That solves targeting and rollout control. It does not create unlimited control-plane capacity.

The generated applications still become objects the control plane must manage. Their resource trees still contribute to cache size and reconciliation work. ApplicationSets can help organize the fleet, but they do not remove the underlying cost of managing every generated application centrally.

Teams should use generators to improve segmentation, governance, and Day 2 operations, while still monitoring the pressure created by the applications they produce.

A clean abstraction can hide growth until the controller starts failing. The ease of creating thousands of applications does not mean the controller can reconcile them cheaply.

Agent-based models attack the right problem

The Argo CD Agent was raised as another direction because it distributes components, including the application controller, across spoke clusters while preserving a central user interface and API.

Conceptually, that attacks the exact bottleneck revealed by the tests. If the central application controller’s memory ceiling is the problem, moving more reconciliation work to agents reduces the amount of state one hub process must carry.

Sveltos used a similar distribution principle in the discussed tests. Lightweight agents ran on managed clusters, and local classification and drift detection reduced pressure on the central control plane.

The challenge is that agent models introduce their own operational questions. Onboarding, mutual TLS, certificate issuance, certificate rotation, trust management, and connectivity direction all matter. One participant described using cert-manager with Red Hat Advanced Cluster Management to manage agent PKI, while noting that teams without that surrounding platform need to plan the certificate lifecycle carefully.

Security architecture also changes. A spoke-initiated connection to the hub is often more comfortable than granting the hub broad access into every managed cluster. But that model depends on a reliable identity and certificate process.

Agent-based architectures can be more scalable, but they are not free. They exchange centralized memory pressure for distributed lifecycle and trust management.

Flux CD adds another useful contrast

Flux CD appeared repeatedly as an alternative worth testing, especially for distributed fleet management. Some teams reportedly moved from centralized Argo CD setups to a dedicated Flux instance per cluster when managing more than 1,000 clusters.

The reasons discussed included scaling behavior, security, extensibility, and the use of OCI as a distribution layer. Git works well for human collaboration, but large numbers of controllers repeatedly reconciling many repositories can expose rate limits and distribution inefficiencies. OCI can be a better fit for packaging and distributing immutable artifacts at scale.

Another operator described preferring Flux CD from an administrative perspective because it behaves more like a Kubernetes-native toolkit and integrates well with workflows and notifications. The same person still valued Argo CD for developers who benefit from its visual interface, RBAC model, and self-service experience.

This is a useful reminder that fleet architecture does not have to standardize on one engine for every problem.

Argo CD may be strong for developer-facing application delivery. Flux CD may suit a toolkit-oriented operating model. Sveltos may handle addon-style fleet rollout efficiently. Combining engines can be more effective than forcing one controller to own every reconciliation pattern.

The architecture should follow workload shape, not product loyalty.

Predict the bottleneck before it becomes an outage

The best prediction model is based on the control plane’s actual work.

Teams should track watched objects per controller or shard, application tree size, reconcile queue depth, reconciliation latency, memory growth, garbage collection pressure, controller restarts, OOM events, Git or OCI fetch behavior, and the effect of rollout waves.

They should also test failure modes that ordinary dashboards may miss:

  • What happens when many clusters reconnect at once
  • What happens when a large addon update triggers fleet-wide reconciliation
  • What happens when Git rate limits or repository latency increase
  • What happens when a controller shard disappears
  • What happens when the hub manages its own configuration
  • What happens when certificate rotation fails for agents
  • What happens when drift accumulates in a partially decentralized model

One comment captured the risk sharply: every cycle in a distributed architecture can become a major incident waiting to happen. A hub that manages itself, a controller that depends on the repository it is reconciling, or a fleet rollout that overloads its own management plane can create cascading failure.

Scale testing should therefore focus on the first domino. The bottleneck may be controller memory, queue depth, Git limits, PKI, network access, or the hub’s own dependencies.

The number of clusters will rarely tell that story by itself.

FAQ

What is the best predictor of a GitOps hub bottleneck?

Watched object count, reconciliation queue depth, controller memory growth, and reconciliation latency are more useful than cluster count alone. They reflect the actual workload carried by the control plane.

Why does the Argo CD application controller run out of memory?

The controller caches application and resource state so it can compare desired and live configurations. As the number and size of watched application trees grow, memory use can rise until the process reaches its limit.

Does Argo CD sharding solve controller OOM problems?

Sharding can delay the problem by spreading applications and cached state across multiple controller replicas. It improves per-process pressure, but it does not necessarily reduce total memory consumption.

Do ApplicationSets improve scalability?

ApplicationSets improve targeting, automation, and fleet segmentation. They can simplify large-scale rollouts, but every generated Application still adds management and reconciliation work.

Is one GitOps controller per cluster more scalable?

It can be, because reconciliation is distributed across the fleet. The tradeoff is greater resource use and lifecycle management on every cluster, which may be unsuitable for small edge environments.

Why are agent-based GitOps models attractive?

Agents move work such as reconciliation, classification, or drift detection closer to managed clusters. This reduces central controller pressure while preserving centralized visibility, but introduces PKI, onboarding, and certificate lifecycle concerns.

Should teams use Argo CD, Flux CD, or Sveltos?

The discussion does not support a universal winner. Argo CD, Flux CD, and Sveltos use different models and may be better suited to different application, addon, security, and fleet-management patterns.

What should teams load test before scaling a GitOps fleet?

They should test large reconciliation bursts, controller restarts, shard failures, repository limits, rollout waves, agent reconnects, certificate rotation, and the behavior of the hub when its own dependencies fail.

CTA

See how infrastructure pressure develops before the management layer becomes the outage. Request an online trial and explore how Sensaka supports clearer operational visibility across complex, distributed infrastructure.