I Planned 10 LLM Evaluation Experiments And Only Ran 1. It Was Enough.

I didn’t set out to write a benchmark paper.

I wanted to answer a much dumber, much more practical question:

“For the stuff I actually do at work, when do I really need a frontier model… and when am I just burning money?”

So I did what engineers do when they want to be serious about this:

  • I designed 10 experiments
  • across real-world agent use cases
  • with a reusable evaluation harness
  • and a budget ceiling so I wouldn’t hate myself later

And then I ran exactly one of them.

It turns out, that one was enough.

The Split-Brain Reality: Work Budget vs Personal Budget

At work, someone else pays for LLM tokens.

  • Frontier models are a Jira ticket away
  • we can afford to run 20-shot experiments on real data
  • “let’s ship and learn” is a reasonable default

At home, it’s different:

  • I can’t run Sonnet or V4 locally
  • my GPU isn’t swallowing 4K context windows at frontier quality
  • every generation is my own credit card

I still want to understand how these models behave on real workloads, not toy prompts. But in my own projects, I have to be surgical about where the money goes.

That tension is what created the frontier experiments program.

I Designed 10 Experiments To Map The Whole Space

On paper, the program was beautiful.

Each experiment was a slice of a bigger question:

“For a fleet of agents doing devopsy things, which model should handle which task, at what effort level, for how much money?”

I designed 10 experiments around that:

  • Adaptive Budget Tuner – what’s the cheapest reasoning budget that still hits the quality bar?
  • Frontier Lite Field Test – can a cheaper model replace an expensive one on my actual workloads?
  • Context Rot Detector – where does long-context recall actually fall apart in real docs?
  • Tool Drift – does tool-calling accuracy degrade when you expose 50–200 tools at once?
  • Code Review Breakpoint – at what PR size and language does AI code review stop being useful?
  • Local Vs Cloud Bench – can a quantized local model get close enough to its cloud counterpart?
  • Cache Hit Maximizer – how much can prompt caching really save on multi-turn agents?
  • Open Weights Head-To-Head – open-weight vs open-weight, not vendor vs vendor
  • Sovereign Agent Lab – can a fine-tuned open model match a frontier model on one narrow task?
  • Apex Threshold Detector – which tasks actually need the most expensive model?

On top of that, I built a reusable evaluation harness, model-compass:

  • 9 real agent tasks (CI diagnosis, incident triage, code review, PR drafting, architecture analysis, etc.)
  • per-task rubrics with binary / exact-match / test-runner / llm-judge scoring
  • a unified adapter layer for Anthropic, OpenAI, DeepSeek, OpenRouter, Together, Kimi
  • auto-computed cost-per-task with USD estimates
  • routing-table generation: “for this task, use the cheapest model that passes”

It was a solid plan.

And then I ran one experiment and stopped.

Why I Chose CI Diagnostics As The One Experiment

I could have started with something “sexier” like architecture reasoning or multi-agent coordination.

Instead I picked CI diagnostics.

Why?

Because that’s closer to what I actually do all day.

My team ships TypeScript, Swift, Kotlin, C++, Java, Go, and BrightScript across multiple repos. Our CI pipelines glue together a lot of first- and third-party tools. When something goes red, the logs are measured in megabytes, not kilobytes. Most failures are in heavy integration tests, and on some days specific areas see 5–20% failure rates just from tool and test flakiness.

As a leader, I wasn’t the one reading every failure. My team was. I invested personally to understand their pain and started building Claude skills that could actually help. The ugly reality was that “CI red → I know the root cause and next step” used to be an hour or more of downloading logs and hunting for patterns. End-to-end, getting a fix to production was often a full day.

With the right automation and AI in the loop, we’ve pushed that down to ~15 minutes to understand what’s going on, and under 2 hours to have a fix merged and back in production in many cases.

Most runs in this experiment started the same way: me squinting at a 5,000‑line GitHub Actions log thinking “this is exactly why I want a model to do this instead.”

If I could learn:

“For CI diagnostics, which model should I actually pay for?”

that would immediately translate into better decisions at work.

So experiment 01 (a precursor to the bigger program) was simple on paper:

  • task: diagnose CI build failures from real logs
  • models: Haiku vs Sonnet
  • data: real GitHub Actions logs from open-source repos (Django, Rails, Node, Pytest, etc.)
  • scoring: did the model identify the right failure domain and give useful, correct steps?

I ran it through the model-compass harness with hard budget caps.

Total cost: $13.53.

$13 looks small, but each experiment has several sub-flavors and with proper sample sizes the full program would have been ~$300+ per experiment. After starting the first flavor of the first experiment, I paired down aggressively and set a $20 hard target to finish the whole thing. That meant cutting sample sizes, dropping sub-flavors, and narrowing scope on the fly.

What This Experiment Is (And Is Not)

Before I go further, it helps to be clear about scope.

What This Experiment Is

  • A focused, personal field test. One DevOps-style task (CI diagnostics) that looks like my real job. Two models, real GitHub CI logs, enough runs to see patterns in quality, latency, and cost.
  • A way to calibrate my own intuition. I wanted to answer questions like: “do I really need the expensive model for CI?” and “when is it safe to default to the cheaper one?”
  • A reality check for my harness. Running this forced me to see which 20–25% of the harness was genuinely reusable, and where experiment-specific scripts (data download, profiles, flavors, summaries) actually carry the weight.
  • Work inside real constraints. Personal budget, no serious GPU farm, and frontier models that do not run locally. The goal was not “cover everything,” it was “learn as much as I can from one well-chosen slice.”

What This Experiment Is Not

  • Not a formal LLM benchmark. I am not running a dozen models across standardized public benchmarks with statistical rigor. I do not have the infrastructure, budget, or full-time evaluation background those teams have.
  • Not a leaderboard verdict. Comparing two frontier models on one CI task does not mean “model A is better than model B overall.” It only says: “for CI diagnostics, with my prompts and data, here is what makes sense for me.”
  • Not the final harness. The original model-compass design assumed a very generic, reusable core. In practice, each DevOps experiment needed its own data pipelines and scripts. This experiment is feeding into a better harness; it is not that final harness itself.
  • Not a replacement for professional evaluation work. Organizations with dedicated eval teams and big budgets are the right people to do broad, formal model comparisons. What I am doing here is much narrower: making one or two better decisions about the models I use for the work I actually do.

The Punchline: The Cheap Model Won

The results:

  • Haiku 4.5 vs Sonnet 5 (via OpenRouter)
  • same prompts, same logs, same scoring rubric
  • ~50 runs across different failure types (DB connection, test failures, build errors, dependencies)

Haiku:

  • 10x cheaper
  • 67% faster
  • statistically tied on accuracy

Sonnet:

  • used more tokens
  • cost more
  • was slower
  • did not clearly outperform on the only metric that mattered: “can you help me fix this build?”

In terms of “should I pay for Sonnet for CI diagnostics?” the answer was:

“Not by default.”

If all you read is that, you might think:

  • “wow, that’s surprising!”
  • “great story about saving money!”

That’s not how it landed for me.

And this part is important:

You should not take this result as “Haiku is better than Sonnet for everything.”

For my specific slice of work:

  • CI log analysis
  • a subset of DevOps-style diagnostics
  • the kind of cases my team actually runs into

Haiku is the model that now makes the most sense to me:

  • it is cheaper than Sonnet
  • it is faster in practice
  • and, based on both public data and this experiment, it is “good enough” or better for these particular tasks

At work, I still reach for Sonnet or Opus a lot — especially for harder reasoning, architecture decisions, or more ambiguous problems.

This experiment just moved one chunk of work (CI / DevOps diagnostics) into the “Haiku by default, Sonnet/Opus when needed” bucket in my head.

There is nuance here. In some runs, Sonnet’s answers were crisper and more coherent. Haiku tended to be more verbose. For CI diagnostics, that verbosity is a feature, not a bug. Haiku is summarizing maybe 1/1000 of the raw logs into something an engineer or a downstream agent can work with. I can live with a small quality gap when a human stays in the loop and gets more context to reason about.

I Wasn’t Surprised By The Cost. That Wasn’t The Lesson.

I went in expecting frontier models to be expensive.

I wasn’t shocked that:

  • CI diagnosis on Sonnet wasn’t “worth it” for this task
  • Haiku looked like the right default for the busywork

I also wasn’t shocked that:

  • a single small experiment burned $13.53
  • a full 10-experiment program would eat hundreds of dollars

Each experiment had several sub-flavors and with proper sample sizes would have been ~$300+ per experiment. After starting the first flavor of the first experiment, I paired down aggressively and set a $20 hard target to finish the whole thing — cutting sample sizes, dropping sub-flavors, and narrowing scope on the fly.

This is exactly why I keep my personal experiments constrained and work experiments more open:

  • at work: bigger purse, more freedom to explore
  • at home: I want to learn from frontier models, but I don’t want a side project to quietly turn into a $300 bill

The actual lesson was something else:

I didn’t need to run all 10 experiments to learn what I came for.

Wide Designs, Narrow Execution

Designing 10 experiments was still worth it.

That process forced me to think clearly about:

  • what “good enough” looks like for different tasks
  • where context length, tool count, and caching actually matter
  • when fine-tuning might be better than just “use the fancy model”

But execution-wise, I realized:

  • most of my day-to-day benefit would come from a few well-targeted insights, not a full laboratory
  • I didn’t need a perfectly mapped frontier of model behavior to make better decisions tomorrow
  • running CI diagnostics as a “typical DevOps scenario” was enough to calibrate my intuition

That single experiment:

  • validated the model-compass harness
  • produced a concrete routing decision for a task I actually care about
  • showed me where the rubric was broken (binary keyword checks vs actual action quality)
  • surfaced the “Haiku vs Sonnet” tradeoff in a way that sticks in my head when I’m choosing models now

The other 9 experiment designs are still there if I ever need them.

But for now, the design work plus one focused execution was enough.

The Messy Part: I Let An LLM Build The Harness

There’s another layer of this that I think matters more than the experiment result.

I asked an LLM to help me build the evaluation harness itself.

It… worked.

Kind of.

What The LLM Did Well

  • cranked out a full comparison script quickly
  • wired up multiple providers with a normalized interface
  • helped me refactor into a cleaner CLI
  • suggested scoring hooks (exact-match, LLM-judge, test-runner) I hadn’t put together yet

What Broke

  • it kept writing "one-off" scripts instead of reusable code
  • path handling bugs, indentation errors, missing fields in JSON/CSV outputs
  • six different scripts ended up archived in a trial/ folder with names like compare_models_v3_final_real_final.py
  • a bunch of API re-runs because of small mistakes, quietly burning a few extra dollars each time

By the third time I fixed a path bug and re-ran the same batch of calls, I could feel the experiment burning money in the background.

The meta-lesson was brutal and simple:

the human has to write the full spec. every field. every output. every edge case.

Once I wrote out:

  • exactly what columns I wanted in the CSV
  • exactly what the routing table YAML should look like
  • exactly how scoring should be attached to rubric dimensions

the LLM was finally useful.

Before that, it was just guessing.

My Harness Design Was More Premature Than I Thought

There was another uncomfortable realization buried in this run.

My original harness design was way more generic than the experiments actually needed.

On paper, model-compass looked like a clean, reusable engine:

  • common task definitions
  • shared tool interfaces
  • one scoring system to rule them all

In practice, the DevOps-style experiments (CI diagnostics, PR review, release-notes generation) all needed their own pipelines:

  1. Large data inputs from public GitHub repos

    Because this is a private project, I couldn’t point the harness at internal repos. I had to:

    • scrape public repos across multiple languages
    • respect GitHub’s rate limits
    • store logs, PRs, and changelists locally

  2. Experiment-specific "automation profiles"

    Each experiment needed its own notion of a "profile":

    • which repo / language / framework
    • what kind of failure or change we were testing
    • what prompt template and tools to use for that slice

  3. Multi-stage scripts per experiment

    The real flow looked more like:

    • script to download and normalize data (rate-limit friendly)
    • script to generate automation profiles
    • script to run one model at a time over those profiles and capture cost + outputs
    • script to compare two models’ outputs per profile
    • script to summarize those comparisons across flavors and sub-flavors

None of that was encoded cleanly in the first version of the harness.

Once the dust settled, I realized we were probably using less than 25% of the original design. Most of the real work lived in experiment-specific scripts.

And that’s… fine.

Nothing is actually throwaway here:

  • the premature parts of the harness told me what not to centralize yet
  • the experiment scripts are concrete proof of what the core should look like
  • the right next step is to fold the good scripts back into the harness once the experiments are over

It’s an unglamorous but useful pattern:

design the generic thing, discover it’s wrong, let the specific scripts show you what the generic layer should actually be.

The other big shift was resisting the “one big script” instinct. DeepSeek V4 Pro’s first attempt was a single script that would download logs, run both models, and spit out final comparisons in one shot. It looked clever, but it made iteration expensive. Breaking it into smaller scripts — “download once and cache immutable logs,” “run a single model over profiles,” “summarize saved outputs” — meant I could fix bugs in one stage without burning API calls in the others. When the summarizer broke, I didn’t have to re-run the models; I just reprocessed the saved outputs. When the downloader broke, I fixed it without touching any LLMs. That separation is what kept this experiment financially and technically sane.

What We Standardized And How TierForge Uses It

Even with just two models, I needed the outputs to line up so comparison wasn’t vibes-based. For each CI run, I forced the model output into a consistent shape: failure domain (a short label like db-connection or test-assertion), a one-to-three sentence root cause summary, evidence pointers (log snippets or line ranges), a coarse confidence bucket (low/medium/high), three to five concrete remediation steps, and a simple verdict on whether this looked like a real issue or something flaky/infra-like.

On top of that, every run captured input/output tokens, cached vs non-cached tokens, total cost in USD, and latency. Scoring and comparison scripts then answered very boring but very important questions: did the failure domain match the labeled truth, was the root cause directionally correct, were the steps actionable and safe, was one model consistently overconfident and wrong, and how much extra did that cost.

From there, I rolled the data up into three layers of summaries: per-flavor (for example, "on DB-connection failures Haiku passed X/Y, Sonnet passed A/B, here’s the cost and latency delta"), per-task across all flavors (overall accuracy, cost to get one "good enough" answer for CI diagnostics per model), and a human-readable decision summary.

This is where TierForge fits. TierForge is an open-source router that turns evidence into YAML routing rules. It wants routing rules backed by real data: for a given use case like ci-diagnostics, what is the primary model, what is the escalation model, and under what conditions do you switch. The experiment gives it exactly that: a concrete "Haiku-by-default, Sonnet-on-these-conditions" rule, plus actual cost-per-task numbers instead of gut feel.

What I Actually Learned (And Kept)

Looking back, here’s what survived and what I keep using.

1. The harness matters more than the paper

I don’t need to publish “definitive bench results” to get value.

What I needed was:

  • a way to run my tasks against different models
  • automatically score the outputs against my quality bar
  • see cost-per-task in dollars, not vibes
  • spit out a routing table I can stick into a config file

That’s what model-compass turned into.

2. One well-chosen experiment beats ten generic ones

CI diagnostics:

  • looks like my actual workload
  • uses real logs, not synthetic prompts
  • exercises error handling, log parsing, explanation quality, and actionability

It’s a better teacher than a grab bag of random tasks I’ll never run in production.

3. My personal budget shapes my learning strategy

Because I can’t justify hundreds of dollars on personal experiments:

  • I design broad (10 experiments)
  • I execute narrow (1 that matters right now)
  • I lean hard on public benchmarks for anything outside my lane

At work, I might run more of the matrix. Privately, I only need enough to calibrate my instincts.

The workload-aware profiles I’m building help a lot here. I am not making model choices in a vacuum; I am making them for specific, well-understood slices of work. That lets me stay inside a tight personal budget without taking a meaningful hit on quality.

4. “Cheap vs expensive model” isn’t just a cost problem

The CI experiment answered a cost question.

The hard part is a behavior question:

will I actually reach for Haiku by default for CI work, or will I still click the fancy model because it feels safer?

I’m still catching myself defaulting to the “nice” model sometimes, even with data in hand.

What I’m Still Figuring Out

I don’t have clean answers here. Just a few open questions I’m carrying around:

  • when you’re on your own dime, how do you decide what’s “enough” experimentation?
  • how do you balance learning from frontier models with not turning your side projects into unpaid benchmark labs?
  • if you’ve actually run cost/quality comparisons, did they change your default model choices… or did you still reach for the shiny one?
  • and if you’ve let an LLM build tools that use LLMs, what did you put in place to keep it from quietly burning your budget?

If you’ve navigated this tension — work budget vs personal budget, curiosity vs cost — I’d love to hear how you handle it.

Yes, the bill matters. But in the teams I talk to, “this is too expensive” often hides other worries: we’re not sure we trust the outputs yet, we don’t feel fully in control of the workflows, or we’re afraid of looking reckless to leadership. I’m trying to go the other way: tighten costs by getting more workload-aware and evidence-driven, instead of tightening costs because we’re scared.

When you think about containing AI costs in your own team, what is really driving it? Is it just the cloud bill, or something deeper — trust, optics, control, or the feeling that you should understand what’s going on before you let a model touch everything?