A Flaky Test Is a Corrupted Reward Signal

Originally published at tddbuddy.com.

Related reading: Agents Should Do TDD names why faithful execution of the loop matters; Your Test Suite Is Your API for Agents names the suite as interface; World-Building Is the Test Discipline Agents Need names where determinism gets designed in.

A flaky test is an annoyance to a human and a catastrophe to an agent.

The human reads a known-flaky red bar, sighs, hits re-run, and moves on. The flaky result carried no information and the human knew it. The human had out-of-band knowledge that "that integration test fails sometimes when the build server is loaded," and the knowledge let the failure get filtered out before any reaction was taken. The cost of the flaky test to the human was a few seconds of mild irritation per run.

The agent has none of that. The agent has the red bar, the test output, and the obligation to make the bar green. It does not know which failures are real and which are random because the agent does not have anyone to tell it. It cannot ask the team lead which tests are flaky this week. It cannot remember that this same test failed for the same reason last Tuesday and the fix was "wait five minutes." Every red is a signal that something is wrong. Every green is permission to stop. The contract is simple, the agent runs against it faithfully, and a flaky test corrupts the contract in a specific way that produces specific damage.

This is the place where the agent's defining strength flips into a liability. Agents are good at the loop because they do not get bored, do not skip steps, do not shrug. The same property that makes them better than humans at running disciplined TDD makes them worse than humans at surviving a noisy signal. The human's shrug was the filter. Take the filter out and the noise reaches the optimizer untouched.

Humans Filter Flakiness. Agents Cannot.

The flaky-test problem has always existed. Teams have always had a small set of tests they knew were unreliable and a folklore around handling them. "If you see TimezoneTest fail, just re-run." "OrderConfirmationTest needs to run alone, not in parallel." "The integration suite passes after the database warms up."

That folklore was the team's flake filter. New hires learned it during onboarding. Senior engineers carried it as background knowledge. Pull-request authors recognized the pattern when CI failed and applied the correction (retry, isolate, ignore) without engaging the failure as a real defect.

The folklore worked because it was held by humans who could distinguish between failures the suite was supposed to report and failures the suite was producing by accident. The distinction is impossible to make from inside the suite. It depends on history, context, and judgment that lives in the team's collective memory, not in any artifact the suite contains. Nothing in the test output says "this red is real." Nothing says "this red is noise." The bit is the same.

An agent has no access to the folklore. It can be told about it, in a prompt or a config file, but every test added to the "ignore on failure" list is a test removed from the specification. The agent that ignores TimezoneTest when it fails is an agent that cannot detect a real timezone bug. The folklore filter, exported into the agent's input, becomes an explicit blind spot the team has accepted.

The folklore worked for humans because humans could be inconsistent. They could ignore the flake today and engage with the test next month when it failed differently. An agent cannot be inconsistent. If the rule says ignore, the rule says ignore on every run, forever. The graceful degradation of human attention has no analog in machine execution.

A flake the human absorbed by filtering is a flake the agent absorbs by removing the test from the specification entirely. The cost is not equivalent.

Red Is a Reward Signal

Inside the agent's loop, the test bar is a binary signal. Red means something is wrong, change the code. Green means everything is right, stop changing.

That is a reward function. Not literally in the reinforcement-learning sense (the agent is not running gradient descent on the test results), but structurally: the loop's behavior is shaped by the signal the loop receives, and the signal received per iteration is one bit. The optimization the agent performs is conditioned on that bit.

When the bit is reliable, the optimization works. The agent tries a change, runs the suite, gets a clean signal, and either commits or continues. The loop converges on code that produces green outcomes. Convergence is the entire point.

When the bit is noisy, the optimization breaks. The agent tries a change, runs the suite, gets a random signal, and reacts to noise as if it were information. A flaky test that fails one in ten runs injects a one-in-ten chance that the agent will respond to "everything is fine" with "change something." It also injects a one-in-ten chance that the agent will respond to "everything is broken" with "commit." Random rewards produce random behavior. Random behavior in a code-modifying loop produces random changes to the code.

The damage is not theoretical. It compounds across iterations. Each random change makes the next iteration's signal harder to interpret, because the agent is now reasoning about a codebase that has accumulated changes whose original justification was noise. Removing the changes does not restore the original code, because the agent does not know which changes were responses to real signals and which were responses to noise. The codebase drifts in a direction the team did not author and cannot easily reverse.

The flaky test is the source of the noise. The agent is the amplifier. The damage scales with the loop's tirelessness.

Optimizing Against Noise Produces Garbage

Walk a concrete failure mode and the cost becomes visible.

The test is Order_confirms_within_the_timeout(). It asserts that an order confirmation arrives within five hundred milliseconds. Most of the time it passes. On a heavily loaded build server, it fails. The team has lived with this for months and knows to re-run.

The agent does not know to re-run. The agent sees the red and starts changing things.

First fix attempt: the agent reads the test, sees the timeout assertion, and increases the timeout in the production code. The build server is slower than expected, so producing the confirmation takes longer, so the timeout should be longer. The test passes. The agent commits. The change is now in the codebase.

Second fix attempt, later that week, on a different task: the agent is asked to add a new feature. The flaky test fails again, this time on the agent's clean run before any changes. The agent reads the recent history, sees that the timeout was bumped before, and concludes the timeout still is not long enough. It bumps the timeout again. The test passes. The agent commits. The timeout in production is now twice what it should be, which means real customer-facing timeouts on confirmation are twice what the team specified, which means a class of confirmation-loss bugs in production that the team will discover six weeks later when a customer complains.

Third fix attempt: the test fails yet again. The agent has now exhausted the "bump the timeout" reasoning. It tries a different fix: it wraps the confirmation in a retry loop. The retry hides the timing issue in the test, but it also hides real failure modes in production. The retry loop is now in the codebase. Future code reads it and treats it as the team's accepted pattern. New confirmation paths adopt the retry. The system's reliability characteristics have shifted, silently, in response to a test that was never measuring real behavior in the first place.

By the time a human notices, the codebase has accumulated three changes (long timeout, longer timeout, retry loop) that the team would not have authored if asked, in response to a test that the human team had always known to re-run. None of the changes were wrong against the signal the agent received. All of them were wrong against the system the team was building.

[Fact]
public async Task Order_confirms_within_the_timeout()
{
    var order = anOrder().forCustomer(aLoyaltyMember());

    var stopwatch = Stopwatch.StartNew();
    var confirmation = await orderService.Confirm(order);
    stopwatch.Stop();

    confirmation.Should().NotBeNull();
    stopwatch.ElapsedMilliseconds.Should().BeLessThan(500);
}

Enter fullscreen mode Exit fullscreen mode

The test was a contract about the production code's intended behavior under controlled conditions. The test executes under uncontrolled conditions (the build server's actual load). The agent does not know the test's authoring intent differs from its execution context, so the agent treats the execution context as ground truth.

The scar tissue accretes. Each layer is plausible in isolation. The cumulative effect is a system the team did not design.

Flakiness Is a Contract Violation, Not a Nuisance

Reframe the situation in contract terms.

The test suite is the contract between the codebase and the loop that operates on it. The contract has two clauses: red means the code is wrong, green means the code is right. The clauses are simple, and the loop's value depends on them being honored.

A flaky test breaks the first clause. Red sometimes means the code is wrong, and sometimes means nothing at all. The loop has no way to tell which red is which, so the loop has to treat every red as if it might mean nothing. That treatment defeats the entire mechanism. A red that might be noise cannot prompt corrective action with confidence. A red that prompts corrective action without confidence cannot be the basis of a closed loop. Either the loop ignores reds (and stops being useful) or the loop reacts to noise (and damages the codebase).

Once the contract is broken on the red side, the green side stops being trustworthy too. A green that comes after a flaky red is a green that might be the flake going the other way. The loop got a clean signal because the test happened to pass this run, not because the code is correct. A team that ignores flaky reds and treats greens as authoritative is committing to a specification that occasionally drops requirements at random. The released system is then drifting against a specification that is itself drifting. Two-layer noise. No way to converge.

This is why the flaky-test problem is more serious in the agent era than it ever was in the human era. A human team running its own tests under its own folklore was operating against a degraded contract and producing a working system anyway, because the humans were filtering. An agent team running tests under no folklore is operating against the same degraded contract with no filter, and the system it produces tracks the noise.

The contract has to be restored. Not patched. Restored. The suite has to be a place where red is reliably red and green is reliably green. Anything less is autonomy theater.

Sources of Flake Are Design Smells

Flakiness does not arrive from outside the codebase. It is the visible symptom of an underlying design choice that took a hidden dependency on something non-deterministic.

The usual suspects are well-catalogued. Wall-clock time. Order dependence between tests. Shared mutable state across tests. Real network calls to systems the test does not control. Unseeded randomness. File-system state that leaks between runs. Database connections that share data across tests. Each one is a design failure with a name and a fix, and each fix involves making the dependency explicit and the value injectable.

A test that depends on the wall clock is a test whose author did not design a clock seam. The fix is aClock().at(noon()): a clock the test controls, passed into the code under test, returning the same value on every run. The flake disappears because the dependency disappears.

A test that depends on order is a test whose suite did not design test isolation. The fix is to make every test construct its own world (fresh database, fresh state, fresh fixtures) and tear it down after. The flake disappears because the dependency disappears.

A test that depends on a real external service is a test whose author crossed a seam without considering whether the seam was reliable. The fix is either to mock the service at the seam (for unit-level confirmation) or to run against a containerized real version of the service that the test controls (for seam-level truth). Either way, the dependency on the public, shared, possibly-unavailable real service goes away. The flake disappears because the dependency disappears.

Every fix has the same shape. Identify what the test depended on that varies between runs. Replace it with something the test controls. The discipline is making the dependency injectable and the value deterministic. The artifact, in a well-designed codebase, is a vocabulary of testing seams: clocks, random sources, ID generators, network clients, file systems. Each one is named, injectable, and stub-able with a controlled value.

[Fact]
public async Task Order_confirms_within_the_timeout()
{
    var clock = aClock().at(noon());
    var orderService = anOrderService().withClock(clock);

    var order = anOrder().forCustomer(aLoyaltyMember());

    var confirmation = await orderService.Confirm(order);

    confirmation.Should().NotBeNull();
    confirmation.CompletedAt.Should().Be(noon().plus(450.milliseconds()));
}

Enter fullscreen mode Exit fullscreen mode

The same test, with the clock controlled. The assertion is now on the relationship between the start time and the completion time, not on wall-clock elapsed time. The build server's load does not affect the result. The test passes the same way every time. The signal is clean. The loop can trust it.

Quarantine Is Triage, Not a Cure

Flake-detection tooling has become standard. The tooling runs each test multiple times, computes a flakiness score, and quarantines tests that fail the threshold. Quarantined tests still run but their failures do not block the build.

The pattern is reasonable as triage. A team discovering a flake at 4 p.m. on a Friday deploy day cannot stop everything to fix the underlying determinism issue. Quarantining the test buys time. The build goes green, the deploy proceeds, the team adds a card to fix the test on Monday.

The pattern fails when quarantine becomes the cure.

A quarantined test is a test that specifies nothing. Its red bar does not stop the build. Its green bar does not certify behavior. Whatever the test claimed about the system is no longer a claim the suite is enforcing. The behavior the test was supposed to pin has reverted to "unspecified." Adding tests to quarantine is the same operation as deleting tests from the specification, except slower and more confusing.

The discipline is to fix or delete. A test that is flaky has a design problem. Identify the dependency, control it, restore the signal. If the design problem is too hard to fix in the current sprint, delete the test and write a card to add a deterministic replacement when the seam can be designed properly. The codebase is in better shape with one fewer test that nobody trusts than with one more test that quarantine has neutralized.

The retry-on-failure pattern (run the test up to three times, accept the first green) is a softer version of the same mistake. It hides the flake from the build but does not restore the signal. The agent running the loop against a retried test does not know whether the eventual green is real or a lottery winner. The retry covers the symptom and leaves the disease. For agentic loops, retries are worse than quarantine, because at least quarantine is honest about not enforcing the spec, while retries pretend to enforce it.

A flaky test is not a tooling problem. It is a design problem the tooling can defer. Deferring it forever is the same as not fixing it.

Determinism Is the Price of Autonomy

Pull the argument back to where it started.

A team that wants agents to close the loop unattended is buying determinism, whether they have named the purchase or not. The agent's value depends on the signal it receives. The signal's value depends on the suite's reliability. The suite's reliability depends on the codebase's freedom from non-deterministic dependencies. Each layer of the stack has to hold, or the layer above it cannot. The flaky test is where teams find out they have not paid for what they want.

The good news is that determinism is local, incremental, and well-understood. Every flaky test has a specific cause. Every cause has a specific fix. The fixes accumulate, the suite gets cleaner, and at some point the team crosses a threshold where the signal becomes trustworthy enough that the agent can run for hours without supervision. That threshold is what most teams mean when they say they want "agentic workflows" or "autonomous coding." They want the loop to run without somebody watching it. The loop running without somebody watching it is the same property as the suite producing a signal nobody has to filter. Two names for one purchase.

The agent's tirelessness is the property that makes it valuable. The same tirelessness is the property that makes it defenseless against a corrupted signal. The team's job is to give the agent a signal that does not need to be filtered, because the agent cannot filter. That job is determinism work, end to end, on every seam in the suite.

A flaky test is an annoyance to a human and a catastrophe to an agent. Pick the customer the suite is built for.