Arthi Arumugam

Every time a new model ships, the same ritual: change the model string, run the agent, read a few replies, they look good, ship it.

The replies are the one part of an agent that almost never breaks visibly. What breaks is behavior. A tool call quietly disappears, an argument drifts, a refund amount loses its decimal point, and the agent keeps talking like everything is fine. I learned this the hard way when a swap made my agent stop calling cancel_subscription while it kept telling users their subscription was cancelled.

With a new frontier model out this week, a lot of model strings are about to change. This is the check I now run before any swap. It takes about 20 minutes and produces a real diff instead of a vibe check.

1. Record a baseline before you touch anything

This is the step you cannot recover later. Once you swap, the old behavior is gone.

Start a recording proxy and point your agent at it, no code changes:

npx whatbroke-cli record --out baseline.jsonl

Enter fullscreen mode Exit fullscreen mode 

OPENAI_BASE_URL=http://127.0.0.1:4141/v1     # openai sdk
ANTHROPIC_BASE_URL=http://127.0.0.1:4141     # anthropic sdk

Enter fullscreen mode Exit fullscreen mode

Run your agent through its real scenarios. Agents are nondeterministic, so run each scenario three times and name the runs refund-flow#1, refund-flow#2, refund-flow#3 (an x-whatbroke-run header per request, or --run). Three samples per scenario is enough to tell flaps from real changes.

Pick scenarios where the agent has to do something: call tools, hit an API, write a record. Pure chat scenarios are where nothing ever visibly breaks, so they tell you the least.

2. Swap the model. Only the model.

Change the model string. Nothing else. If you also want to tweak the prompt for the new model, do that as a second swap with its own diff, otherwise you will never know which change caused what.

3. Record the same scenarios again 

npx whatbroke-cli record --out swapped.jsonl

Enter fullscreen mode Exit fullscreen mode

Same scenarios, same names, three runs each.

4. Diff 

npx whatbroke-cli diff baseline.jsonl swapped.jsonl

Enter fullscreen mode Exit fullscreen mode

Read it top down:

  • breaking findings first: dropped tool calls, runs that now fail, outputs that vanished. Any of these means the swap is not a drop-in.
  • changed findings next: argument drift is the sneaky one. The tool still gets called, but with different args. Check every one by hand.
  • The flap rate tells you whether a finding is real. 3/3 means it happens every time. 1/3 on something your baseline also flapped on is just your agent being itself, and the diff demotes those automatically.
  • Cost and latency move on every swap. Regressions past a ratio get flagged (defaults 1.5x latency, 1.25x cost).

5. Keep it in CI 

npx whatbroke-cli diff baseline.jsonl current.jsonl --fail-on breaking

Enter fullscreen mode Exit fullscreen mode

Exit code 1 on breaking changes, --md for a report you can drop into a PR comment.

Already swapped without a baseline?

If your agent runs behind Langfuse, LangSmith, or anything emitting OTel GenAI spans, you already have the baseline, you just have not diffed it yet. Export last week's traces and this week's:

npx whatbroke-cli import last-week-export.json --run baseline
npx whatbroke-cli import this-week-export.json --run swapped
npx whatbroke-cli diff last-week-export.whatbroke.jsonl this-week-export.whatbroke.jsonl

Enter fullscreen mode Exit fullscreen mode

The tool is deterministic, fully offline, MIT licensed, and your traces never leave your machine: https://github.com/arthi-arumugam-git/whatbroke

For a real example of what a swap changes while the replies all look fine, I ran the same agent on a 3x smaller model and wrote up what actually changed.

If you run this before your next swap and it catches something, I would genuinely love to hear what it was.