Ask an engineer to list their agent's failure modes and you'll hear about hallucinations, wrong tool calls, and bad JSON. Ask about time and you get a shrug. Yet the single most common thing a production agent does when it goes wrong is not fail loudly — it just takes too long. It loops. It retries a flaky tool. It waits on a model call that never streams a first token. And your eval suite, which runs after the fact on whatever output eventually showed up, grades it green.

This is the blind spot: we treat latency as an SRE dashboard concern, divorced from correctness. But for an agent, a deadline miss is a correctness failure. A summary that arrives 90 seconds late is often worse than no summary — the user already left, the downstream job already timed out, the retry already double-charged them. Time belongs in your evals, and it belongs at the very bottom of the stack.

Time is Tier 1 evidence

If you've followed the tier doctrine behind agent-eval, you know evidence ranks on an independence axis — from evidence the agent can't forge, to opinion it shares a substrate with — not a cost axis. Three tiers:

  • Tier 1 — externally observable proof the agent can't fake: valid JSON, the file exists, the code compiled, tests passed, it finished within the deadline, the output is non-empty.
  • Tier 2 — statistical signal against a baseline the agent didn't author: embedding similarity to the task, length and repetition, whether the diff actually changed anything.
  • Tier 3 — model-as-judge: a shared-substrate opinion. A signal, never a verdict.

Notice where "finished within the timeout" sits. It's Tier 1, right next to "valid JSON." That's not an accident. A wall-clock deadline is the most independent signal you have — the agent cannot argue with a stopwatch, cannot reason its way past it, cannot forge it. Physics grades this one. There is no more incorruptible ground truth in your whole eval suite than "the clock ran out."

And crucially, Tier 1+2 are your real-time gate: deterministic, ~$0, fast enough to sit in the hot path and block a run. A timeout check is the purest expression of that — it's already in the hot path by definition. Tier 3, the judge, is the opposite: metered, slow, non-deterministic, offline-only. You would never put a model-as-judge on the critical path to decide whether a response was fast enough. The judge can't even see the clock.

What "graded green" actually looks like

Here's the trap. Most eval harnesses receive (input, output) and score the output. The output is whatever the agent finally returned — so by construction, timing information has already been discarded before grading begins. The eval literally cannot see that the run blew its deadline, because it only exists once the run is over.

You have to grade the run, not the return value:

type RunResult<T> =
  | { status: "ok"; value: T; elapsedMs: number }
  | { status: "timeout"; elapsedMs: number; deadlineMs: number }
  | { status: "error"; elapsedMs: number; error: string };

async function withDeadline<T>(
  work: (signal: AbortSignal) => Promise<T>,
  deadlineMs: number,
): Promise<RunResult<T>> {
  const ctrl = new AbortController();
  const started = Date.now();
  const timer = setTimeout(() => ctrl.abort(), deadlineMs);
  try {
    const value = await work(ctrl.signal);
    return { status: "ok", value, elapsedMs: Date.now() - started };
  } catch (err) {
    const elapsedMs = Date.now() - started;
    if (ctrl.signal.aborted) {
      return { status: "timeout", elapsedMs, deadlineMs };
    }
    return { status: "error", elapsedMs, error: String(err) };
  } finally {
    clearTimeout(timer);
  }
}

Enter fullscreen mode Exit fullscreen mode

Now the eval gate is trivial and deterministic — a Tier 1 check that runs in microseconds:

function tier1TimingGate(run: RunResult<unknown>): { pass: boolean; reason?: string } {
  if (run.status === "timeout") {
    return { pass: false, reason: `deadline miss: ${run.elapsedMs}ms > ${run.deadlineMs}ms` };
  }
  if (run.status === "error") {
    return { pass: false, reason: `crashed after ${run.elapsedMs}ms` };
  }
  return { pass: true };
}

Enter fullscreen mode Exit fullscreen mode

A timed-out run never reaches Tier 2 or 3. There's nothing to embed, nothing to judge — the correct answer that arrives too late is not a correct answer. You short-circuit, you block the run, you fall back. No model was asked for its opinion, because no opinion was needed.

Deadlines are per-budget, not per-agent

The mistake I see next is a single global timeout. Deadlines are contextual: a background reindex job can take ten minutes; an interactive chat turn has maybe three seconds before the user perceives a stall. The deadline is a property of the call site, not the agent. Thread it through as part of the task contract, and let every tool step inherit and decrement a shared budget — so a tool that eats 80% of the budget leaves the model no room to actually respond, and that is a gradeable event too.

You can't grade time you didn't record

All of this assumes you actually captured elapsed time per step — and here's where the eval half of the story needs its other half. agent-eval scores and gates the output; it can only enforce a Tier 1 timing gate if something recorded the timing. That's the job of trace capture. AgentLens instruments the agent's trajectory — every model call and tool step, with resolved inputs, raw outputs, and start/stop timestamps — so "finished within the deadline" is a fact you can read off an unforgeable trace the agent didn't get to author, not a number you hope somebody logged.

This pairing is the whole point. The trace is the substrate Tier 1+2 grade against: because AgentLens records when each step started and stopped independently of what the agent claims it did, your timing gate has real ground truth. An agent can hallucinate that it "responded quickly." It cannot edit the timestamps in a trace it didn't write. That's the difference between an eval that catches the 80% of boring failures — stale, crashed, empty, too slow — deterministically at Tier 1+2, and a dashboard of green checkmarks that quietly hides every run that limped across the finish line a minute late.

Reserve the judge for the subjective 20% — tone, helpfulness, "did this actually answer the question" — and label it opinion, not evidence. But the stopwatch? That's evidence. Put it first.