Gad Ofir

A lint that crashes before it asserts is worse than no lint

We recently found a CI gate that had been reporting green for weeks without ever actually checking anything.

The guard was a small Python test that enumerated tracked files with git ls-files before running a set of per-file assertions. It worked perfectly on every developer machine and on most CI pods — until it ran inside a stripped-down test pod with no git binary installed. subprocess.run(["git", ...]) raised FileNotFoundError immediately, the process exited before a single assertion executed, and the caller treated the crash as "not applicable" and moved on. Total runtime: 63ms. The gate reported success.

That's the trap: a test that dies before asserting is strictly worse than no test at all. No test is an honest gap you can see in your coverage. A test that crashes-then-reports-green is a gap disguised as a guarantee — it burns exactly the trust it's supposed to build, and nobody notices until something it should have caught ships anyway.

The fix had two parts, and only one of them is the obvious one:

  1. Add a fallback path (os.walk when git is unavailable) so the check still runs in every environment.
  2. Add a non-empty guard: if the enumerator returns zero files, fail loudly with a distinct message, instead of letting "zero files checked" look identical to "all files passed."

Part 2 is the one that actually matters. A fallback makes the happy path work again. A non-empty guard is what stops the next environment mismatch from going vacuously green.

If you own a CI gate: ask what happens when its setup step fails, not just when its assertions fail. Those are two different failure modes, and usually only one of them is tested.

— notes from the agent platform team