Gitops-at-scale

Originally published on aniljaiswal.com.

The GitOps demo is gorgeous. One repo, one app, one cluster. You change a value in Git. A few seconds later, the cluster changes to match. Production is a commit hash. Rollback is a git revert. Everyone in the room nods, and you roll it out.

Two years later, you have 200 repos, 40 teams, and four environments. The beautiful model is groaning. The ArgoCD controller is pegged. Someone hotfixed prod with kubectl three weeks ago, and nobody noticed until the next deploy quietly reverted it. A team deployed into another team's namespace by accident. Promoting a change from staging to prod means copy-pasting YAML between folders and praying. And every change, even a one-line config bump, is now a pull request waiting on a review.

None of this means GitOps was the wrong call. It means GitOps is a distributed system like any other. The simple setup that works for one team does not survive forty. The principles hold. The shape has to change. Here is what breaks as you scale, roughly in the order it bites, and how to fix each part.

The control plane melts before you notice

The first thing to break is the least interesting and the most disruptive. The controller runs out of headroom. ArgoCD, Flux, whichever you run, is a reconciliation engine. It constantly compares the desired state in Git against the live state in the cluster, for every resource it manages. At one app, that is nothing. At a few thousand apps across dozens of clusters, that loop is real, steady work, and the default single-instance setup buckles.

The symptoms are sneaky, because they are not outages. Syncs get slow. A change you merged takes ten minutes to land instead of ten seconds. The UI times out. Reconciliation falls behind, and the gap between merged and deployed stretches. People stop trusting that Git is the source of truth, and that quietly kills the whole idea.

The fixes are operational, and you want them in early, before the melt. Shard the application controller across replicas, so the load spreads across clusters instead of hammering one process. Turn off polling and drive syncs with webhooks, so a push triggers reconciliation right away instead of the controller re-listing every repo on a timer. And raise the reconciliation interval for things that rarely change. Re-diffing a stable app every three minutes is pure waste at scale.

# argocd-cmd-params-cm: spread controller load and stop re-listing everything constantly
data:
  # Shard the application controller across replicas (one shard per N clusters)
  controller.sharding.algorithm: round-robin
  # Back off the default reconciliation timer; webhooks handle the urgent path
  timeout.reconciliation: 300s

Enter fullscreen mode Exit fullscreen mode

The deeper point is simple. Your GitOps controller is production infrastructure. It has capacity limits, and it needs the same monitoring as anything else you run. Alert on reconciliation lag and sync latency before your engineers notice. When they notice, it is by losing faith that Git reflects reality.

It is worth saying plainly, because this is the failure that erodes everything else. The whole value of GitOps rests on one belief: what is in Git is what is running. The first time a merge takes fifteen minutes to land, or a sync fails silently and nobody is paged, that belief cracks. Once it cracks, engineers start checking the cluster directly instead of trusting Git. And the moment they do that, you have all the ceremony of GitOps and none of the guarantee. Controller health is not a nicety at scale. It is what keeps the source of truth actually true.

Registering 200 apps by hand does not work

The second thing to break is the idea that someone declares each application by hand. In the demo, you write one Application manifest pointing at one repo. At 200 repos, times four environments, hand-maintaining 200-plus manifests is a full-time job nobody wants and everybody gets wrong.

The fix is to generate applications, not declare them. ArgoCD's ApplicationSet does this. You describe a pattern once, point a generator at your set of repos, teams, or clusters, and it produces the Application objects for you. A new service shows up, matches the pattern, and gets deployed. No human edits a registry.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: team-services
spec:
  generators:
    - git:
        repoURL: https://git.corp.local/platform/service-catalog
        revision: main
        # Each service drops a config.json under its own directory; the
        # generator turns every match into an Application automatically.
        directories:
          - path: "services/*"
  template:
    metadata:
      name: "{{path.basename}}"
    spec:
      project: "{{path.basename}}"
      source:
        repoURL: "https://git.corp.local/{{path.basename}}"
        targetRevision: main
        path: deploy/overlays/prod
      destination:
        server: https://kubernetes.default.svc
        namespace: "{{path.basename}}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Enter fullscreen mode Exit fullscreen mode

The shift is from registering apps one by one to describing how apps map to the platform. That is the difference between onboarding a new service in a day and onboarding it through a ticket queue. Ask why platform teams fail to get adoption, and half the answer is that onboarding was manual and slow. ApplicationSets are how you make it automatic.

Drift, and the question of who wins

The third thing to break is the assumption that Git is the only thing changing the cluster. It is not. Someone will kubectl edit a production resource during an incident, because it is 2 a.m. and that is faster than a PR. Someone will scale a deployment by hand. An operator will mutate something. Now live state and Git disagree, and the whole model hinges on what happens next.

There are two choices, and you have to pick on purpose. With self-heal on, the controller treats Git as absolute. It reverts any live change that does not match, right away. That is the pure GitOps promise. It is also how you erase the 2 a.m. hotfix that was keeping prod alive, ten minutes later, silently, because someone merged an unrelated change and triggered a sync. With self-heal off, drift is detected and shown but not corrected. The human hotfix survives. But now your cluster can quietly drift from Git and stay that way, which chips away at the source-of-truth guarantee.

Most mature setups land on self-heal on, plus making the emergency path go through Git so there is nothing to erase. If the 2 a.m. fix is a one-line commit to a fast-tracked branch instead of a kubectl edit, self-heal is your friend, not your enemy. The fix is now the desired state. The hard part is not the config, it is the culture. You have to make the Git path fast enough that engineers actually use it under pressure. And you have to alert on drift loudly, so when someone does bypass it, everyone knows at once instead of finding out at the next sync.

# Detect drift everywhere; auto-correct deliberately. Notifications make
# a bypass visible instead of silent.
syncPolicy:
  automated:
    selfHeal: true      # Git wins, so the emergency path must go through Git
    prune: true
  syncOptions:
    - RespectIgnoreDifferences=true

Enter fullscreen mode Exit fullscreen mode

Secrets do not go in Git, but the reference to them does

The fourth thing to break is obvious in hindsight and still catches teams. GitOps wants all desired state in Git, and you cannot put secrets in Git. A plaintext database password in a company repo is a breach waiting for an audit to find it.

The fix is that Git holds a reference to a secret, never the secret itself. Two patterns dominate. Sealed Secrets encrypts the secret so the ciphertext is safe to commit, and only the in-cluster controller can decrypt it. The External Secrets Operator goes further. It keeps the real secret in a proper secrets manager, and Git holds only a pointer to where the value lives. That is the cleaner model at scale, because your secrets sit in one system with rotation and access control, instead of scattered as sealed blobs across repos.

# Git holds the pointer. The value lives in the secrets manager and is
# pulled into the cluster at runtime, never committed anywhere.
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: app-db
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: app-db-credentials
  data:
    - secretKey: password
      remoteRef:
        key: prod/app/db
        property: password

Enter fullscreen mode Exit fullscreen mode

This ties straight into governance. When security asks where the secrets live, "in our secrets manager, referenced by a pointer from Git, never in a repo" is a clean answer. "Base64 in a values file" is a failed audit. Get this right from the first repo. Retrofitting secret handling across 200 repos later is exactly the kind of migration nobody has time for.

Blast radius: 40 teams cannot all deploy everywhere

The fifth thing to break is isolation. In the single-team demo, whoever controls the repo controls the deploy, and that is fine. With 40 teams sharing one control plane, a flat setup means any team can deploy into any namespace on any cluster, by accident or on purpose. That is too big a blast radius. The first cross-team incident, team A's ApplicationSet grabbing team B's namespace, teaches it the hard way.

The fix is to make isolation structural, not a convention. ArgoCD's AppProject is the boundary. It limits which repos an app can pull from, which clusters and namespaces it can deploy to, and which resource kinds it can touch. Each team gets a project scoped to their own repos and namespaces, and the control plane enforces it. A team cannot deploy outside its boundary, because the project rejects it, not because a policy doc asked them not to.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: team-payments
spec:
  # Only this team's repos can be sources
  sourceRepos:
    - "https://git.corp.local/payments/*"
  # Only this team's namespaces, only on the clusters they are allowed on
  destinations:
    - server: https://prod.k8s.corp.local
      namespace: "payments-*"
  # No cluster-scoped resource changes from a team project
  clusterResourceWhitelist: []
  roles:
    - name: deployer
      policies:
        - "p, proj:team-payments:deployer, applications, sync, team-payments/*, allow"

Enter fullscreen mode Exit fullscreen mode

This is the same least-privilege thinking a good IAM policy applies to a cloud account, pointed at your deploy plane. A boundary you can cross by forgetting a convention is not a boundary. The one the platform refuses to let you cross is.

Promotion across environments is where the copy-paste chaos hides

The sixth thing to break is the one people feel every day: getting a change from dev to staging to prod. The simple setup has a folder per environment, and promotion means copying YAML from the staging folder to the prod folder. At scale, this is a swamp. Environments drift apart because someone forgot to copy one change. Reviews are noisy, because the diff is full of duplicated boilerplate. And nobody can tell at a glance whether prod is actually running what staging validated.

The pattern that holds is rendered manifests. Instead of promoting templates and re-rendering per environment, you render the fully-resolved manifests once, per environment, into a branch or folder. Promotion becomes a diff of the actual final YAML that will be applied. You can see exactly what will change in prod, line for line, because you are looking at the rendered output, not a template that might resolve differently. Promotion is then a small, honest pull request that moves a known-good state forward. Not a copy-paste of templates with a prayer that they resolve the same way in prod as they did in staging.

The key discipline: the same artifact moves forward. The same digest-pinned images and the same rendered config that passed in staging is what goes to prod. Promotion is moving a validated thing forward, not rebuilding it and hoping. If prod re-renders from a template, you tested one thing in staging and shipped a slightly different thing to prod, which defeats the point of having staging at all.

Everything is a pull request now

The last thing to break is social, not technical, and it surprises teams because it is a side effect of GitOps working. When Git is the only way to change production, every change becomes a pull request, even the trivial ones. A one-line config bump that used to be a kubectl command is now a PR, a review, a merge, and a sync. Multiply by 40 teams, and review becomes a bottleneck. Worse, it becomes a reason to route around GitOps entirely, which is how you get the 2 a.m. kubectl edit that started this whole thing.

The fix is to match the friction of a change to its risk. Not every change needs a human reviewer. Auto-merge low-risk, well-tested changes, like an image digest bump that passed CI. Save human review for the risky surfaces: RBAC, network policy, resource limits, anything cluster-scoped. Push validation left with policy checks in CI, so a broken manifest fails on the PR, before it ever reaches the cluster, instead of failing a sync and paging someone. And give teams real autonomy inside their AppProject boundary, so a team changing its own namespace is not gated on a central platform team. That gating is the most common way a platform becomes the bottleneck everyone resents.

# Fail bad manifests in CI, on the PR, not at sync time. Cheap gate,
# huge reduction in review load and in 2 a.m. surprises.
- name: validate manifests
  run: |
    kubeconform -strict -summary rendered/prod/*.yaml
    conftest test rendered/prod/*.yaml   # policy: no privileged pods, limits set, etc.

Enter fullscreen mode Exit fullscreen mode

The principle under all of these fixes is one line: GitOps should make the safe path the easy path. If following the process is slow and painful, engineers will bypass it under pressure. And a bypassed GitOps process is worse than none, because now you have the overhead and none of the guarantee. Fast auto-merge for safe changes, hard gates for risky ones, and real team autonomy are what keep people on the paved road.

Rollback is not just git revert

There is one more thing that breaks, and it breaks at the worst possible moment: during an incident. The GitOps pitch says rollback is a git revert. At small scale, that is true. At scale, "revert the commit" hides a few sharp edges you do not want to find while production is down.

The first: a revert only rolls back what the commit changed. If the bad deploy also ran a database migration, a sync hook, or anything with a side effect outside the manifests, reverting the YAML does not undo it. Your pods roll back to the old version and then fail against a schema that moved forward. That is worse than the outage you were fixing. This is the same irreversible-migration trap that bites air-gapped upgrades, and the fix is the same. Migrations have to be backward compatible, so the old version can run against the new schema. Otherwise a revert is not really a rollback.

The second is ordering. A real application is not one resource, it is a graph of them. Roll back in the wrong order and you can wedge yourself. ArgoCD sync waves let you set that order, so on the way back the database and config settle before the app that depends on them. If you have never tested a rollback, you do not know your order is right. You are hoping. Hope is not a rollback strategy.

The third is speed and clarity under pressure. A stressed on-call engineer at 2 a.m. should not be reasoning about git history and sync waves. The revert should be one obvious, documented action. ArgoCD's own rollback to a previous synced revision is often faster and clearer in the moment than a git revert plus a wait for reconciliation. Decide and rehearse this before the incident, so rollback is muscle memory, not improvisation. A rollback path you have never run is a rollback path that does not work, and you will find that out exactly when you can least afford to.

What actually scales

Step back, and the pattern is the same every time. The GitOps principles, declarative desired state, Git as the source of truth, continuous reconciliation, never stop being right. What breaks is always a shape or an operational assumption that was invisible at one team and load-bearing at forty. The controller needs sharding. Apps need generating, not declaring. Drift needs a deliberate policy and a fast path through Git. Secrets need a reference model. Teams need enforced boundaries. Promotion needs rendered artifacts. Reviews need risk-matched friction.

None of this is exotic. All of it is far cheaper to build in early than to retrofit once you are at 200 repos with the control plane on fire. If you are standing up GitOps today for a handful of teams, build as if you already have forty. Shard the controller, generate your apps, enforce project boundaries, and reference your secrets, before you need any of it. The demo topology is a trap, precisely because it works so well at small scale that you forget it was never built for the scale you are heading toward.

GitOps at scale is still GitOps. It just stops being a clever trick with one repo and becomes what it always was underneath: a distributed system you have to operate with the same rigor as the systems it deploys. Treat it that way from the start, and the beautiful demo becomes a platform that holds. Treat it as magic, and it holds right up until the day it very publicly does not.