A change to a service that consumes from Kafka is not validated until it has run against the real system: real partitions, real consumer groups, the events other teams produce. Coding agents sharpen the requirement. An agent iterating on a consumer needs to publish an event, observe what its change did downstream, read the failure and rerun, many times per change and in parallel with every other agent’s loops. That only works if a realistic environment is available on demand, for every change.
“Testing Kafka consumers is hard. You have to deploy Kafka, producers, consumers, and other service dependencies to validate flows e2e. There is a better pattern.”
For synchronous services, the architectural pattern we enable at Signadot delivers exactly that: lightweight ephemeral environments on a shared cluster. Instead of duplicating the stack per change, you run one shared environment continuously deployed from main and deploy only the changed service next to it. Test requests carry a tag, each hop routes tagged traffic to the version under test, and everything else falls through to the shared stable versions.
That model rides on request routing, and request routing stops at the first Kafka topic. Nothing between a producer and its consumers picks a destination per message, so the tag that isolates a change everywhere else in the system stops meaning anything at the async hop.
Extending the model across that hop takes three mechanisms: a routing key that rides the message, producers that stamp it, and consumers that filter on it. The rest is detail, and the detail is where the pattern succeeds or fails.
Why request routing stops at the topic
A synchronous call passes through something that can choose a destination per request: a sidecar, a proxy, a client library. A Kafka record is written once and then pulled by every subscribed consumer group at its own pace and its own offset. There is no delivery-time decision point to hook.
“The tag that isolates a change everywhere else in the system stops meaning anything at the async hop.”
Deploying a changed consumer next to the stable one fails both ways available to you. Put it in the stable consumer group and partition assignment splits the topic between the two versions arbitrarily. Give it a separate group and both versions process every message, doubling every side effect.
The contamination runs in both directions. The version under test executes unreviewed code against other people’s messages and writes the results into a shared downstream state. The stable consumer processes your test events, so you cannot tell whether the new code handled them. And because an async flow leaves no request chain to follow, the contamination is hard to even see.
The standard workarounds trade the wrong things. A duplicated cluster per environment means brokers, connectors, schema registry, and seed data drifting from production from day one. Per-environment topics push the cost into topic lifecycle automation and per-environment rewiring of every producer and consumer config. A single-node throwaway broker from Testcontainers is the right tool for unit-level consumer logic and says nothing about the shared system.
The fix is a routing key that rides the message
Give each test deployment an opaque routing key, k7 in the examples below. Tag the initiating test request with that key at the edge. On the synchronous legs, the key travels in OpenTelemetry baggage, and you need only the context propagation part of OpenTelemetry, not tracing.
At publish time, copy the key from the request context into the record headers. Headers, not payload: the message body stays untouched, and consumers can filter without deserializing. Every mainstream broker has an equivalent slot: message attributes in SQS and Pub/Sub, headers in AMQP.
Put the copying in shared code, not in each service. OpenTelemetry producer instrumentation can do it, or a thin internal wrapper around the standard client, so a service publishes exactly as before and propagation stays uniform across teams.
The key names the test, not the workload
This is the detail that makes the pattern compose. The key identifies the test context that caused a message, not the deployment that produced it. A stable producer serving a request tagged k7 emits records tagged k7.
“The key identifies the test context that caused a message, not the deployment that produced it.”
That is how a consumer change gets tested in isolation. No test version of the producer exists; the stable producer stamps your key, and only your consumer version picks it up.
On the consumer side, restore the key from the headers into the processing context before the handler runs, so every downstream call and every further publish carries it onward. Isolation holds only while that chain is unbroken. Flows without an originating request need the key seeded explicitly, and change data capture (CDC) is the sharpest case: the connector has no request context, so the key goes into a metadata column and the connector copies it into headers.
Consumers filter, and exactly one version processes each message
Every consumer version subscribes to the shared topic and receives everything. The decision is per message, in a should-process gate ahead of the handler. A test consumer processes a message only when the header key matches its own test context. The stable consumer processes untagged traffic, plus tagged messages whose key no live test consumer claims.
The fallback clause does two jobs. A producer-only test still works, because stable consumers handle its tagged messages downstream. And another test’s key never reaches your version under test. The net effect: exactly one consumer version processes each message.
Consumer groups keep the versions from interfering at the offset level. Each test consumer joins a fresh group named after its deployment, tracks its own offsets, and starts at the latest offset so it does not replay the topic’s backlog. The stable group’s commits are untouched. The group is created with the test deployment and deleted with it.
Consumers learn which keys are claimed from a small in-cluster mapping service that holds key-to-deployment state. Poll it and cache the result, or stream updates from it. The gate consults the cached map on every message.
A validation run, end-to-end
Here is the loop as an agent runs it. The agent changes the payment consumer and deploys the modified version alongside the stable one, with key k7, its own consumer group, and a registration in the routing map.
The agent then sends a request tagged k7 at the edge, or produces a tagged event directly. The stable order producer stamps k7 into the records it publishes. On the topic, those records sit interleaved with everyone else’s traffic. The k7 consumer processes them, the stable consumer skips them, and the effects flow downstream still carrying k7.
The agent asserts on downstream state, reads whatever failed, patches the consumer, redeploys and reruns. Between iterations, only the changed deployment moves. When the change merges, deleting the test deployment removes the group and the map entry, and nothing else in the system changes. Dozens of these loops run on one cluster at once, each seeing real partitions, real rebalancing, and real traffic from producers other teams operate.
A few things to keep in mind
The routing key is not the partition key. Each version sees its messages in partition order, but a test that depends on strict ordering across interleaved tests and production traffic will not get it. Batch consumers must split each batch by key before processing. The routing map cache is stale for a few seconds around deployment create and delete, which is exactly when a first message tends to arrive.
Some cases still call for duplicated infrastructure: when the broker configuration is itself the thing under test, and when compliance requires hard tenant separation. Everything else gets a different cost model, a test environment that amounts to the changed services plus a consumer group.
There is also a second approach worth knowing. Instead of routing on a shared topic, you can spin up ephemeral topics and point the producers and consumers at them for the life of the test. That swaps the routing work for reconfiguration: every producer and consumer in the flow has to be rewired to the new topic names when the environment comes up, and switched back when it goes away. Our resource plugin framework is built for exactly this. A plugin provisions the topics on sandbox creation, tears them down on deletion, and passes the connection details into the sandboxed workloads.
A concrete check for your own system: trace how the last schema change to a shared topic was validated before merge, and how long it waited for an environment where that was possible. If the answer involves a calendar, that topic is where your isolation ends and where your agents will queue. We built Signadot around this pattern, and it plugs into the Kafka implementation you already have.
Group Created with Sketch.
0 Comments
Log in to join the conversation.No comments yet. Be the first to share your thoughts.