Silent Data Loss in Celery + RabbitMQ on Azure Container Apps: A Queue Registration Post-Mortem

Hello friends! I'm back with another real-world debugging story from the trenches.

If there's one goal every backend engineer should have,, it's building systems that fail loudly instead of quietly dropping data while returning 202 Accepted. Today, we're diving into a production incident that took days to track down-not because the code was wrong, but because cloud infrastructure and container entry points had a silent disagreement.

Grab your coffee, and let's go hunting for ghost queues.

Introduction

One of the first lessons every engineer learns moving from a monolith to microservices is that consistency stops being free. In a monolith, a write either commits inside your transaction or it doesn't; there's one database, one process, one truth. Split that same write across services and you trade that guarantee for eventual consistency: the write will happen, just not necessarily right now, and not necessarily in the same process that accepted the request.

Message queues are the workhorse of that tradeoff. A typical pattern looks like this:

Client → API (producer) → Message Broker → Worker (consumer) → Database

Enter fullscreen mode Exit fullscreen mode

The API accepts the request, publishes a message, and immediately returns 202 Accepted. Somewhere downstream, milliseconds later, usually, a worker process picks up that message and actually does the work. This is a genuinely good pattern: it decouples slow work from request latency, it lets you scale producers and consumers independently, and it gives you a natural retry boundary if something downstream is briefly unavailable.

It also has a failure mode that almost never shows up in tutorials: what happens when the consumer isn't actually listening, and nothing tells you?

This is the story of a bug that took days to run down, precisely because every individual piece of evidence pointed somewhere reasonable and wrong. The application code was correct. The message broker was healthy. The container was running, healthy, and logging normally. And yet, for a specific subset of operations, every single write vanished - accepted, acknowledged, never persisted, and never once producing an error anywhere in the stack.

The root cause, when we finally found it, wasn't in our code at all. It was a single stale field on a cloud platform resource that nobody had thought to check.

The Architecture, Briefly

The system in question is a fairly standard FastAPI + Celery + RabbitMQ setup, deployed as three services on Azure Container Apps:

• A Frontend/API service that accepts HTTP requests and enqueues Celery tasks for anything that writes data.
• A Worker service running Celery, consuming from RabbitMQ and writing to Postgres.
• A Data service behind the worker, doing the actual persistence.

Celery's queueing model is simple on paper. Each task gets a name and, usually, its own dedicated queue:

# worker/tasks/order_tasks.py
from celery import shared_task
from worker.main import app

@app.task(
    name="orders.tasks.register_order",
    bind=True,
    queue="orders.tasks.register_order",
)
def register_order_task(self, command):
    order_service = self.resolve(IOrderService)
    return order_service.register_order(command)

Enter fullscreen mode Exit fullscreen mode

Declaring @app.task(...) registers the task in Celery's Python-level task registry the moment the module is imported. That's step one. Step two, the one that actually matters for whether messages get processed, is telling the worker process which queues to consume from, at startup:

# docker/worker.dockerfile
ENTRYPOINT ["celery", "-A", "worker.main", "worker", \
            "--loglevel", "info", \
            "-Q", "orders.tasks.register_order,orders.tasks.edit_order,orders.tasks.import_order_row"]

Enter fullscreen mode Exit fullscreen mode

That -Q flag is the entire contract. If a queue name isn't in that comma-separated list, the worker never opens a consumer for it, even if the task is fully defined, fully imported, and fully capable of running. Celery and RabbitMQ don't complain about this. Nothing complains about this. A producer publishing to an unconsumed queue succeeds silently, because RabbitMQ (correctly, by design) auto-declares the destination queue on first publish regardless of whether anyone is listening.

That single asymmetry, task registration and queue subscription being two independent things that happen to usually move together, is where this entire incident lived.

The Symptom

The bug looked exactly like a straightforward, if odd, application defect:

POST /consumables
{"name": "Test Item", "code": "TEST001"}

→ 202 Accepted
{"task_id": "8f3a...", "data": {"id": "e8942ca9-...", "name": "Test Item", "code": "TEST001", ...}}

Enter fullscreen mode Exit fullscreen mode

A plausible-looking response, complete with a freshly generated ID. Everything about it looked like success.

GET /consumables/e8942ca9-...
→ 404 Not Found

Enter fullscreen mode Exit fullscreen mode

The record simply never existed. Not eventually. Not after a retry. Not after ten seconds, or ten minutes. This affected exactly four entity types out of seventeen in the system, consistently, at 100% loss, every single write, every time, forever.

No exceptions. No 5xx responses anywhere in the chain. No entries in a dead-letter queue (there wasn't one). No metric crossed a threshold, because there was no metric measuring "messages published to a queue with zero consumers." From the outside, four specific operations across the whole API had quietly stopped doing anything, and the system was structurally incapable of telling anyone.

Chasing Ghosts: What We Ruled Out

Given the symptom - accept, then silently vanish- the natural first suspects are the ones every distributed-systems postmortem reaches for:

• A duplicate-write race condition. Ruled out: this happened on the very first, uniquely-coded write, no concurrency involved.
• RabbitMQ policies applying unexpected arguments to specific queues (this broker was shared across several unrelated services, so a stray wildcard policy from someone else's team was a real candidate). Checked directly in the RabbitMQ management UI - zero policies configured, user or operator.
• Per-vhost or per-user resource limits silently rejecting new queue declarations once some threshold was hit. Checked - none configured.
• Stale queue/exchange metadata - the theory that these specific queues had been declared at some point with incompatible arguments (different durability, different type) and were now permanently rejecting redeclaration. This one felt promising, until we tried the most direct possible test: restarting the entire RabbitMQ broker. A full broker restart wipes any transient inconsistency and forces every queue to be redeclared from scratch by whoever connects next. The bug survived it, completely unchanged. Theory dead.
• The application code itself. This is the one that eventually broke the case open, precisely because it kept coming back clean. We took the exact -Q queue list from the deployed Dockerfile, spun up a disposable RabbitMQ container locally, and started the worker against it with the identical code:

docker run -d --name repro-rabbitmq -p 5680:5672 rabbitmq:3.13-management-alpine

celery -A worker.main worker --loglevel=info -Q "$(cat queue_list.txt)" &

grep "exchange=" worker_boot.log | wc -l
# → all 67 expected queues bound cleanly, zero errors

Enter fullscreen mode Exit fullscreen mode

Every single queue, including the four that were failing 100% of the time in production, bound perfectly against a clean, local broker running the exact same code. Twice we ran this experiment, from two different angles, and twice the code came back innocent.

At this point we had eliminated the broker, the policies, the limits, and the code. That doesn't leave many places left to look, which was itself the clue we were missing.

The Breakthrough: Ask the Container What It's Actually Running

Every diagnostic tool we'd used up to that point application logs, celery inspect registered, the RabbitMQ management UI, even the Dockerfile itself- describes what should be true. None of them describe what the running process was actually invoked with. There's a difference, and on a container orchestration platform, that difference is exactly where a bug can hide indefinitely.

Azure Container Apps supports exec'ing directly into a running replica:

az containerapp exec \
  -n worker-service \
  -g my-resource-group \
  --command "/bin/bash"

Enter fullscreen mode Exit fullscreen mode

And once inside, there is exactly one place that tells you the unfiltered truth about how PID 1 was started:

cat /proc/1/cmdline | tr '\0' '\n'

Enter fullscreen mode Exit fullscreen mode

The output was not what the Dockerfile said it should be.

# What the Dockerfile's ENTRYPOINT specifies:
celery -A worker.main worker --loglevel info --autoscale 5,1 -Q orders.tasks.register_order,orders.tasks.edit_order,...,orders.tasks.import_order_row  [67 queues total]

# What /proc/1/cmdline actually showed:
celery --app=worker.main worker --loglevel=info --autoscale=10,1 -Q orders.tasks.register_order,orders.tasks.edit_order,...  [39 queues total]

Enter fullscreen mode Exit fullscreen mode

Different flag syntax (--app= vs. -A), a different autoscale value, and a drastically shorter -Q list, missing every import-row queue and every queue for the four failing entities entirely. This wasn't a formatting quirk. It was a completely different command than the one baked into the currently-deployed image.

Root Cause: The Silent Platform-Level Override

Azure Container Apps lets you set an explicit command and args override directly on the Container App resource, independent of whatever ENTRYPOINT/CMD is baked into the image. If that override is set, it wins unconditionally, regardless of what the image itself specifies.

At some point, months earlier, someone had set this override, most likely when the app was first provisioned, before several of the entities (and their import queues) in this codebase existed. Since then, every deploy had used a CI/CD step that looked like this:

az containerapp update \
  --name worker-service \
  --resource-group my-resource-group \
  --image "myregistry.azurecr.io/worker-service:${GIT_SHA}"

Enter fullscreen mode Exit fullscreen mode

Only --image. Never --command, never --args. Every single deploy since the override was first set had faithfully shipped new code - and then silently ignored it, because the platform kept running the old, frozen command instead of the new image's own entrypoint. The Dockerfile was a complete, accurate, up-to-date source of truth for a command that had never actually run in production.

Confirming and fixing this took two steps. First, read the override back out in full, since a value this long is too easy to corrupt by hand-editing a form field in a portal UI:

az containerapp show \
  -n worker-service -g my-resource-group \
  -o yaml > worker-app.yaml
# edit the containers[0].args block directly
az containerapp update \
  -n worker-service -g my-resource-group \
  --yaml worker-app.yaml

Enter fullscreen mode Exit fullscreen mode

Then - and this is the part worth remembering- a redeploy alone does not fix a bug like this. New revisions of a Container App still boot the same overridden command unless the override itself is changed. Every restart, every scale event, every new revision from that point forward had been re-running the identical stale command. The fix isn't "redeploy"; it's "stop overriding the image's own entrypoint," or at minimum, keep the override in lockstep with the Dockerfile by hand, which is itself a second, ongoing source of risk.

Data Loss and the Eventual Consistency Trap

Here's the part that matters beyond this one platform quirk: this failure mode is a direct, sharp-edged consequence of how "eventual" consistency actually works.

Celery's default publisher behavior auto-declares the destination queue when a message is sent, whether or not any consumer exists (task_create_missing_queues, on by default). That's normally a convenience; you don't have to pre-provision every queue by hand. But it also means a producer with a perfectly correct queue name will happily publish into the void forever. RabbitMQ doesn't reject the message. It doesn't expire it by default. It just sits there, Ready, unconsumed, for as long as nothing is listening, which in this case turned out to be months.

# RabbitMQ management UI, Queues tab
Name                                   Ready  Unacked  State
orders.tasks.register_order            5      0        running

Enter fullscreen mode Exit fullscreen mode

Five messages, sitting there, 0.00/s deliver rate, indefinitely. That number is the whole bug, visualized: five real user actions, accepted by the API, and then never acted on again.

"Eventual consistency" is a promise that a write will eventually be reflected - it is not a promise about how long eventually is allowed to be, and it says nothing at all about what happens if the thing that's supposed to make it happen structurally never shows up. When the consumer side of an async pipeline is silently missing, "eventually" quietly becomes "never," and the system has no vocabulary for telling you the difference. A 202 Accepted and a permanently-lost write look identical from the caller's side, because in both cases the only promise actually made was "we'll get to it", never "we did it."

A related discovery made the same point from a different angle. While chasing this, an unrelated-looking bug turned up: certain list endpoints intermittently returned an empty 204 No Content even though the records genuinely existed. The real cause was a data-mapping layer raising a validation error on a NULL field, which the internal HTTP client silently converted into a generic "not successful" response with no status code preserved, so what was actually a 500 on one internal hop surfaced to the end user as an innocent-looking "no data here." Different bug, same underlying lesson: any layer that swallows the specific reason for a failure and reports a generic, plausible-looking result is doing your future self a disservice. Silent degradation is worse than a loud failure, because a loud failure gets fixed.

What We'd Do Differently

A few concrete takeaways, in rough order of how much they'd have saved us:

1. Verify the running process, not just the source of truth. /proc/1/cmdline (or your platform's equivalent, docker inspect, kubectl get pod -o yaml, and check the actual container spec, etc.) tells you what's actually executing. A Dockerfile, a Helm chart, an IaC template - these are all descriptions of intent. Only the running process tells you the truth, and on any platform that supports a command/args override independent of the image, that intent and that truth can diverge invisibly.
2. Avoid dual sources of truth for a container's startup command. If the Dockerfile's ENTRYPOINT fully specifies how to start the process, don't also maintain a platform-level override of the same thing. Pick one. If a platform-level override is unavoidable for some reason, treat any deploy that doesn't also touch it as suspect.
3. Instrument the gap, not just the endpoints. An unconsumed-message-count alert on your broker (Ready > 0 and deliver/get ≈ 0 for more than N minutes, per queue) would have caught this in minutes instead of days. Application-level health checks and 200-OK dashboards are blind to this entire failure class by construction, the API was never unhealthy; the worker was never crashing. The problem lived entirely in the space between two healthy-looking components.
4. Write a queue-parity check. A five-line script that statically extracts every queue= from your task decorators and diffs it against whatever's actually passed to -Q at deploy time would have caught this before it ever reached production, and would catch the next instance of "we added a new entity and forgot to wire its queue" too, which, notably, is exactly what happened a second time later in this same incident, for twelve other queues nobody had thought to check.
5. Don't trust "it's already correct" claims made against source files alone. More than once in this incident, a piece of code being correctly committed was mistaken for that code actually running in production. Those are different facts, and conflating them is exactly the trap that let this bug hide as long as it did.

Why This Bug Was So Difficult to Find

Looking back, what made this incident particularly difficult wasn't that any individual component was broken-it was that every component appeared to be working correctly when viewed in isolation.

At different stages of the investigation, each layer gave us evidence that suggested the system was healthy:

• The API accepted requests and returned 202 Accepted as expected.
• Producers successfully published messages to RabbitMQ.
• RabbitMQ accepted and stored every message without errors.
• Celery itself was healthy and running normally.
• Azure Container Apps reported healthy containers and successful deployments.
• Application logs showed no exceptions or crashes.
• Our deployment pipeline completed successfully, giving us confidence that the latest code was running.

None of those observations were incorrect. In fact, every single one was true.

The problem was that the worker process was consuming only a subset of the queues because of a stale Azure Container Apps command override. Every component behaved exactly as it had been configured to behave—the configuration itself was the problem.

This is what made the incident so deceptive. We naturally tend to investigate individual systems in isolation:

Is RabbitMQ healthy?

Is Celery running?

Did the deployment succeed?

Is the application code correct?

The answer to every one of those questions was yes.

The real failure existed between those systems, in the assumptions that connected them.

That's one of the hardest lessons I've learned about distributed systems: many production incidents aren't caused by a single failing component; they emerge from the interaction between components that are each behaving correctly according to their own configuration.

When that happens, traditional health checks, deployment success indicators, and application logs can all tell a perfectly consistent—but incomplete—story. Solving the problem requires shifting your focus from individual services to the contracts and assumptions that connect them.

Conclusion

None of the individual technologies here were at fault. Celery did exactly what it was told. RabbitMQ did exactly what it was told. The application code, when we finally proved it beyond doubt, was correct the entire time. The failure lived in the seam between the deployment platform and the container image, a place none of our existing tooling was looking, because none of our existing tooling had a reason to distrust it.

That's the real lesson of eventual consistency in a microservice architecture: decoupling a producer from a consumer buys you resilience against the consumer being slow. It buys you nothing against the consumer being silently absent, and the more hops your architecture has, the more places that absence can hide. The fix, in the end, wasn't a code change at all. It was asking the one question none of our monitoring had ever thought to ask: is the thing that's running actually the thing we think we deployed?

If you've hit something similar- a queue, a topic, a subscription that looked perfectly configured everywhere except in the one place that mattered- I'd genuinely like to hear about it in the comments.

Final Thoughts & Conversation

Distributed systems give us incredible scale and decoupling, but they trade away obvious failure modes. When your health checks, application logs, and deployment pipelines all report 100% green, the hardest bugs to solve are the ones hiding in the seams between your platform configuration and your code.

Have you ever ran into a bug where an infrastructure override or an auto-declaring broker silently ate your data? How do you handle queue parity and container command parity in your CI/CD pipelines? I'd love to hear your horror stories and prevention strategies in the comments below!

---

A Quick Side Note:
I regularly write about backend performance, queue architectures, system reliability, and real-world debugging post-mortems. If you enjoy deep-diving into distributed systems, defensive engineering, or API architecture, I'd love to connect and chat!

You can find me on LinkedIn, follow my latest technical thoughts on Twitter/X, or check out my open-source projects and previous articles on GitHub.

Thanks for reading, and see you in the next post-mortem!