I did not get to this opinion from a whitepaper.

I got to it because I am building an open-source project around the problem, and the build keeps arguing back.

I am working on AgentEval Forge right now, an OSS evaluation lab for agents. The original idea sounded straightforward enough: scenario packs, adversarial cases, trajectory scoring, regression tracking, cost and latency analysis. In my head, it was going to be a more serious, more structured version of evaluation work I had already done elsewhere.

That was the first mistake.

I had already spent time building model-eval and workload-eval style systems. I knew what it meant to compare models for real tasks, how to design rubrics, how to think about speed versus cost, and how quickly a neat score can become misleading if the harness is weak. I assumed agent evaluation would be an extension of that same world.

The deeper I got into AgentEval Forge, the less true that felt.

Model evaluation asks whether the answer is good.

Agent evaluation has to ask whether the system behaved well enough to trust.

Those are not the same question.

The comparison that made it click

A clean model-eval setup is something I know how to reason about. You have an input, an output, and some way of scoring quality. Sometimes that is exact match. Sometimes it is a rubric. Sometimes it is an LLM judge. Sometimes it is a benchmark harness. However fancy the setup gets, the center of gravity is still pretty stable: did the model produce a good answer for this task?

That is already hard enough. I have seen enough brittle scoring and false confidence to know that even model evaluation can go wrong in ways that look scientific. A rubric can reward the wrong thing. A binary check can miss something obviously useful. A benchmark can feel objective while still pulling you toward the wrong optimization target. So I am not pretending model eval is solved.

But with agents, the shape of the problem changes.

The thing being evaluated is no longer just an answer. It is a workflow. It is a sequence of decisions. It is tool choice, retries, intermediate state, cost, recovery behavior, and sometimes policy adherence. The moment you let a system do more than respond once, the final output stops being the whole story.

That sounds obvious when you say it plainly. In practice, I think a lot of teams still use answer-scoring habits on systems that have already moved past that category.

What changed for me while building AgentEval Forge

The real shift happened when I tried to define what AgentEval Forge would actually have to score.

At first, I thought the answer would be simple: run scenario packs, score the outputs, compare versions, done.

But the more I looked at real agent behavior, the more that started to feel naive. An agent can produce a decent final answer while doing a lot of things badly on the way there. It can choose the wrong tool first and recover by luck. It can loop more than it should. It can spend five times the tokens a better path would spend. It can take a risky step that a human reviewer would never approve, then still back into something that looks acceptable at the end.

If I only score the final answer, I might mark that run as a success.

That is the trap.

The path matters.

That is the sentence I keep coming back to while building this. The path matters just as much as the endpoint, and sometimes more, because the path is where cost, safety, and trust actually live.

That challenge shows up in very practical ways. I am not just trying to score whether an agent got the answer right. I am thinking about cases where it picked the wrong tool first, passed the task only because it recovered late, crossed a boundary it should not have crossed, spent too many steps on something simple, or regressed only inside one narrow scenario family while the average score still looked fine. That is the kind of mess that makes agent evaluation feel much more like systems work than answer grading.

I had already seen hints of this in earlier work

This was not the first time I had felt the edges of the problem.

In earlier harness and field-study work, especially when running systems against real workloads instead of toy examples, I kept finding failures that would never have shown up in a neat prompt-eval setup. That experience made me much less impressed by demo-path confidence.

A system can look strong in a controlled evaluation and then become erratic as soon as the environment gets irregular. Real repos. Real logs. Real commit histories. Real naming mess. Real ambiguity.

That is where edge cases show up. That is where tool misuse shows up. That is where “technically correct” and “safe to ship” start to drift apart.

I think that history is part of why AgentEval Forge took the shape it did in my head. I was not trying to build a leaderboard generator. I was trying to build something that feels closer to release discipline for agents.

Not: did this run look good?

More like: if I change this system, what got better, what got worse, what got more expensive, and what became riskier even if the top-line score improved?

That feels much more like engineering than benchmarking.

The five things I now think agent evaluation has to cover

The more I think about it, the more I come back to five dimensions.

First, you still need final correctness. If the agent does not solve the task, the rest only matters so much.

Second, you need trajectory quality. How did it get there? Was the sequence of decisions sensible, efficient, and stable? Or did it succeed in a way you would never want repeated in production?

Third, you need to evaluate tool behavior. Did it choose the right tools? Did it overuse them? Did it take expensive or risky actions unnecessarily? Did it rely on accidental recovery?

Fourth, you need safety and policy adherence. A successful output reached through an unsafe path should not count as a clean win.

Fifth, you need cost, latency, and regressions. Did the new version get slower? More expensive? More erratic? Did it improve accuracy while making operational behavior worse? Did it just change style and still get celebrated as progress?

That last one matters a lot to me because I think it is where teams fool themselves most easily. They see difference and call it improvement. Those are not the same thing.

Why I do not think the current eval ecosystem is enough

I am not arguing that the current eval ecosystem is useless. Far from it.

The LangChain piece on evaluation-driven development gets a core thing right: once you see failures in production, those failures need to feed offline evals, and every change should be tested against them. That loop is healthy. It is exactly the kind of discipline I want more teams to adopt.

And from a different angle, Birgitta Böckeler's writing on agentic coding captures something else I think matters: the most dangerous failures often live in longer feedback loops. Maintainability. Team friction. Brute-force fixes. Misdiagnosis. Overbuilt solutions. Those costs do not always show up in the immediate output, but they absolutely show up later.

That is why I do not think the issue is that existing tools are bad. I think the issue is that many of our evaluation habits were formed in a prompt-and-answer world. Agent systems are messier. They behave more like orchestrated software than isolated text generation. Once you accept that, your evaluation system has to grow up too.

Even the eval can lie to you

This is the part I do not want to over-neaten.

Your evaluation setup can be wrong in ways that feel rigorous.

I have already seen enough of that in model-eval and harness work to be wary of any system that sounds more certain than it really is. You can undercount useful behavior. You can make your binary checks too brittle. You can optimize for what is easy to score rather than what actually matters.

So now the challenge has two layers. The agent is hard to evaluate, and the evaluation system itself becomes another system you have to design carefully, calibrate honestly, and distrust a little.

That is not a reason to avoid serious evaluation. If anything, it is the opposite. It means evaluation has to be treated like product and engineering work, not just like reporting.

Why I am still optimistic about building this

For all of that, I do not feel cynical about the problem.

If anything, building AgentEval Forge has made me more convinced that this is worth doing well. I want scenario packs. I want adversarial cases. I want trajectory scoring. I want regression tracking that can tell me which change made the system more useful and which one only made it look cleaner in a demo.

I want an evaluation system that helps answer a release question, not just a research question.

Did this version get better?

Better how?

At what cost?

And what got riskier even if the headline score went up?

That is the kind of question I want tooling to help with.

The bigger thing I think teams are about to learn

I suspect a lot of teams are still doing with agent evaluation what many teams did with AI-assisted coding more broadly: borrowing habits from an earlier problem and hoping they scale.

Sometimes they do.

Often they do not.

And then the bill shows up later, in regressions, strange production behavior, expensive paths, unsafe tool use, or a team that keeps changing the system without ever being able to say whether it actually improved.

That is why this topic feels worth writing about while I am still in the middle of building the OSS for it. The build itself is sharpening the argument for me. Every design decision keeps pushing me toward the same conclusion: model evaluation asks whether the answer is good. Agent evaluation has to ask whether the system behaved well enough to trust.

That is a harder problem. I think it is also going to become one of the defining engineering problems of serious agent work.

I will write more about this once the OSS is public. Some of these challenges are already turning into concrete design decisions in AgentEval Forge, and in the coming weeks I want to share which parts we solved cleanly, which parts stayed messy, and how the tradeoffs changed once the code had to work as a real system instead of an idea. I will share the GitHub repo publicly when that is ready.

Open for discussion

This is still an active build and an active line of thinking for me, so I would genuinely like pushback on it.

  • Where did answer scoring stop being enough for your agent work?
  • Are you evaluating the path yet, or still mostly the final output?
  • What has been hardest to score well: tool use, regressions, safety, cost, or something else?
  • And the big one: are we building evaluation systems that improve release confidence, or just better ways to feel rigorous?