Rotating the Hostile Seat: A Six-Round Adversarial Design Review Before Hardening an Agent

Originally published on hexisteme notes.

I was about to harden a new agent whose whole job is to turn "should I adopt this library, model, or tool" into a deterministic, auditable verdict instead of a vibe — gates, grades, falsifiers, a learning ledger. Before trusting it with that job, I wanted a design review nobody could dodge. My default pattern was "ask my main coding assistant to look it over," which has the same structural problem as a same-family writer reviewing its own writing: builder and checker share the same blind spots. So this time three roles — Questioner, Answerer, and adversarial Verifier — rotated through three reviewer groups in every possible assignment, across six rounds. Three roles into three groups is exactly six permutations, and I used all of them, so no group ever sat as the permanent judge.

The setup: eight targets, six dimensions, three groups

The system under review had eight discrete pieces worth judging, pulled from its own codebase rather than picked after the fact: identity and boundaries (what separates a verdict-making agent from a plain fact-gathering one), four type-level invariants blocking an unverified claim from being laundered into a confirmed fact, five deterministic scoring gates that only score fact-labeled evidence, the grade decision and hard-gate demotion logic built on those gates, automatic derivation of the conditions that would prove a verdict wrong, a provenance parser with a host whitelist for fact-grade sources, a learning ledger checking whether its own confidence is honestly calibrated, and the CLI/bus/config surface a human touches. Each got judged on six dimensions: interesting to judge, useful downstream, complete against its own spec, coherent with its docs and siblings, reliable — reproducible, tested, falsifiable — and actually serving the system's purpose.

Going in: eight open verdicts on record, zero recorded outcomes, 677 lines of tests. Zero outcomes matters more than it sounds — a learning ledger with nothing in it yet is the cheapest moment to catch a calibration bug, since every future outcome joins against whatever's already sitting in there.

Three groups reviewed: my own fleet of other agents — a coordination hub plus domain workers I normally run for research, narrative, and real-estate decisions, borrowed here purely as reviewers; my main coding assistant, reasoning directly with no tool fan-out; and an external multi-model verification pool — a cross-model proxy reaching several independent model families, including a large open-weight model, plus a symbolic-math engine, a theorem prover, and a couple of code-analysis tools.

Why rotate instead of picking a reviewer

Three roles times three groups gives six possible assignments, and I fixed all six before round one — nobody learned their next assignment as they went; the full table existed up front. That's what stops a group from steering later rounds toward a friendlier seat.

Across the six rounds, each group asks the questions twice, defends twice, and attacks twice. That's the whole mechanism. A fixed reviewer has one failure mode: whatever it can't see, nobody sees, forever. A designer grading its own fix has another: no incentive to find its own flaws. Rotating fixes both — not by being smarter, but by guaranteeing the seat most likely to catch a mistake is eventually filled by someone who didn't make it, and whichever group designed a fix later sits in the chair whose only job is to attack it.

What it actually caught

Six rounds against eight targets produced seven confirmed defects, one claim checked and found correct (a symbolic-math engine confirmed a worried-about scoring formula was just the standard Beta-binomial posterior mean, no fix needed), and several accusations a second party disproved outright, including one hallucinated claim about a counter that didn't exist anywhere in the code. The defects that generalize past this one codebase:

F1's twin, F2: the scoring gates' coefficients — multipliers, day-count normalizers, dependency and breaking-change weights — lived only in code, undocumented and unreproducible by hand.

F6 is worth slowing down on — it's the whole thesis in miniature. Round 1 flagged the mismatch. Rounds 2 and 3, different groups in different seats, converged on a fix: when the hard-demotion threshold fires, quietly promote status to "partial" and expose the reason. That direction was proposed, selected, and cross-checked across three rounds. Round 4 put my own fleet agents in the attack seat, specifically to break whatever had just been agreed — and it did: promoting a hard demotion to "partial" dresses up a loud failure (a fatal input defect) as a valid partial pass, close to the opposite of what a demotion means. Inside that round, one internal viewpoint tried to back its case with simulated percentages, and the group's own fabrication check caught them as invented and kept them out. The three-round consensus got reversed; what shipped instead was smaller and safer — expose the demotion reason on the result, leave status alone.

F7 and F8 share a quieter lesson. F7 — a bug that would have silently corrupted the learning ledger's calibration data — surfaced because round 1's attack seat went looking for exactly this instead of confirming what the defend seat claimed. F8 came from a different overreach: round 2 assigned my research worker to the question-asking seat only, and instead of asking, it grepped the code for evidence first — which is how a freshness check that looked implemented but was never called got caught. Nobody wrote a rule requiring the questioner to verify its own questions; rotating it into a seat with something to prove was enough.

The verification pool needed its own degrade path

Round 1's attack seat was supposed to include the external multi-persona council tool I normally reach for in verification rounds. It didn't respond. Falling back to a cross-model proxy should have been simple, except its usual route — an API aggregator reaching a newer hosted model — also failed with an expired key. What produced round 1's findings was a second fallback: a local runner reaching a large open-weight model, gpt-oss 120B, plus a symbolic-math engine for the one claim needing computation instead of judgment. Three layers deep before anything answered, in the very first round.

The review didn't pause to fix the verification stack. It substituted, logged what it substituted and why, and kept going. A review process that only works when every tool in it is up isn't resilient, it's just untested. The degrade path is part of the design, not an incident.

The same pattern recurred later: whenever a round needed that external council tool and it wasn't available, my own fleet agents' coordination hub substituted for it directly rather than stalling. Verification infrastructure going down mid-review was routine enough to need a designed fallback, not a one-off workaround.

Honest costs: six rounds is a lot of turns

I won't pretend this is free. Six rounds, three participants swapping seats each time, is eighteen passes over the same eight targets before anything gets fixed — a real cost for a solo builder, and running it on every change would be its own kind of dysfunction. It's overkill for anything small and reversible: a copy edit, a config default that's easy to flip back, a UI tweak nothing downstream depends on. If being wrong costs you one revert, a full rotation is theater, not rigor.

It paid for itself here because of what this system is: type invariants, scoring gates, and a calibration ledger everything downstream trusts without re-checking. F7 is the clearest argument for the expense — a silent ledger corruption wouldn't announce itself, it would just make every future verdict's confidence a little more dishonest, untraceable to a same-day re-evaluation months earlier. F6 makes the same case differently: a demotion that quietly reads as a partial pass is exactly the bug a consumer only discovers by trusting the wrong field, possibly after acting on it. The rule I'm taking forward: spend the rotation on code other automated decisions will trust unchecked, skip it on anything a human looks at first.

The recipe

What I'd actually reuse from this, stripped of the specifics:

1. Enumerate the review targets and judging dimensions up front —
   pulled from the real code, not decided as you go.
2. Fix the full role x group permutation before round one. If it
   can change, someone finds a way to dodge the hostile seat.
3. Log every round append-only, including refuted suspicions — a
   rejected accusation is still evidence the check happened.
4. Run at least one attack round AFTER a fix has multiple groups'
   agreement, not only before — that's the round most designs never
   get, and the one that caught an accepted fix that was wrong.
5. Convert confirmed defects into surgical fixes and regression
   tests before writing anything new. Log what's deliberately not
   fixed yet, and why.

Enter fullscreen mode Exit fullscreen mode

It needs the schedule decided before the review starts, and a standing refusal to let the group that built something also be the last word on whether it's safe.

More notes at hexisteme.github.io/notes.