Our test suite had a flake we couldn't pin down. A test would pass alone, pass in its own file, then fail when the full suite ran. Re-run it and it passed again. Classic state leak.

The culprit turned out to be the thing we thought was protecting us: pytest-xdist.

The Problem

xdist spreads tests across persistent worker processes. Those workers stay alive and pick up file after file. That is great for spawn overhead and terrible for isolation. Any module-level dict, any ContextVar, any singleton that one test file mutates rides along into the next file that lands on the same worker.

We had ~17k tests across ~850 files. The state that leaked was almost always module-level: a registry populated at import time, a cache that never got reset, an env-var read once and memoized. None of it was the test author's bug. It was the worker being reused.

You can chase this with fixtures forever. We tried. autouse reset fixtures, monkeypatched globals, teardown hooks. Every one of them is a patch over the real issue: the interpreter is shared when it shouldn't be.

The Fix

Give every test file its own fresh Python interpreter. Run one python -m pytest <file> per file, with bounded parallelism. No persistent workers, no shared process, nothing to leak through.

The whole pool is a ThreadPoolExecutor handing out subprocess launches:

from concurrent.futures import ThreadPoolExecutor
import os, subprocess, sys, time

def run_one_file(file, pytest_args, repo_root, file_timeout):
    cmd = [sys.executable, "-m", "pytest", str(file), *pytest_args]
    proc = subprocess.Popen(
        cmd,
        cwd=repo_root,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        text=True,
        env=os.environ,
        # POSIX: head of its own process group so we can kill the
        # whole tree atomically. Windows: maps to CREATE_NEW_PROCESS_GROUP.
        start_new_session=True,
    )
    try:
        output, _ = proc.communicate(timeout=file_timeout)
        return file, proc.returncode, output
    except subprocess.TimeoutExpired:
        kill_tree(proc)
        return file, 124, "(file timeout; process tree killed)"

with ThreadPoolExecutor(max_workers=os.cpu_count()) as pool:
    futures = [pool.submit(run_one_file, f, args, root, 300) for f in files]
    # ThreadPoolExecutor.__exit__ blocks until all are done

Enter fullscreen mode Exit fullscreen mode

That's the core. A semaphore-gated Popen pool in about 60 lines, versus xdist's loadfile/loadscope modes, --max-worker-restart, and an internal control plane we didn't need.

Why Per-File and Not Per-Test

The obvious question: if isolation is the goal, why not one process per test?

Because the math kills you. Process spawn is roughly 250ms. Per test that is 250ms x 17,000 = ~70 minutes of pure spawn overhead before a single assertion runs. Per file it is 250ms x 850 = ~3.5 minutes, which fits the CI budget.

And per-file buys you the isolation boundary that actually matters. Cross-file module state was the entire flake source. Intra-file state is the test author's responsibility, and a fresh interpreter per file draws the line exactly where the bug lived.

Gotchas

The part that bit us hardest was killing timed-out processes. A test can spawn a uvicorn server or an async runtime as a grandchild. proc.kill() only takes out the immediate child. The grandchildren reparent to PID 1 on Linux, or get adopted by services.exe on Windows, and they leak, holding ports and eventually starving the runner.

You have to kill the whole tree. And you have to capture the process group id before the leader exits:

pgid = None
if sys.platform != "win32":
    try:
        pgid = os.getpgid(proc.pid)   # capture NOW
    except (ProcessLookupError, PermissionError):
        pgid = None

Enter fullscreen mode Exit fullscreen mode

Here is the trap. Once the leader process is reaped, os.getpgid(proc.pid) raises ProcessLookupError even though grandchildren in that group are still alive. If you wait until cleanup time to look up the pgid, it is already gone and you kill nothing. Capture it right after Popen, then SIGKILL the group:

def kill_tree(proc, pgid=None):
    if proc.pid is None:
        return
    if sys.platform == "win32":
        # taskkill walks the recorded ppid chain, works after the root exits
        subprocess.run(["taskkill", "/F", "/T", "/PID", str(proc.pid)],
                       stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=10)
    else:
        os.killpg(pgid, signal.SIGKILL)   # atomic, whole group

Enter fullscreen mode Exit fullscreen mode

We reached for psutil first. It does not help here: in the happy path the root is already reaped, so psutil.Process(pid) can't find it, and grandchildren reparented to PID 1 aren't reachable by a tree walk either. The platform-native primitives (process groups on POSIX, taskkill /F /T on Windows) handle both the alive-root and dead-root cases without the extra dependency.

The other cost is honest: per-file spawn is slower than a warm xdist worker on a single file. We ate that on purpose. A test suite you can trust in 3.5 minutes beats a flaky one in 2.

One More Thing: Retry the File, Not the Test

Fresh interpreters cut the flakes way down, but a truly timing-sensitive test can still misfire once. Instead of a blanket rerun plugin, the runner retries a failed file exactly once in a brand new subprocess. If the retry passes, the file counts as green, but it gets printed in a FLAKY summary with both attempts' output.

That summary matters. A pass-on-retry is not "handled," it's a bug you now have a reproduction window for. Silently swallowing it is how a suite rots back into the flaky state you just escaped. Surfacing it keeps the pressure on to fix the underlying timing assumption.

def run_with_retry(file, args, root, timeout, retries=1):
    for attempt in range(retries + 1):
        f, rc, out = run_one_file(file, args, root, timeout)
        if rc == 0:
            if attempt > 0:
                mark_flaky(f)   # green, but reported loudly
            return f, rc, out
    return f, rc, out

Enter fullscreen mode Exit fullscreen mode

What about you?

If you're fighting flakes that only show up in the full suite, check whether your runner reuses workers before you write another reset fixture. How are you drawing your isolation boundary?