We run one LLM call on every agent trace we ingest: it reduces the trace to a short, searchable digest. Because it runs on every trace from every customer, it is the most frequent LLM call we make, and its cost is the one that grows fastest with our usage.

So we tested whether a cheaper model could do it. Sonnet 4.6 writes that digest today at about $3.45 per 1,000 traces, and plenty of models cost a fraction of that. We put four of them against Sonnet on a frozen set of 50 traces. The one we shipped runs at about a thirtieth of the cost with search returning the same traces, and the two that looked cheapest on the price sheet both failed.

The harder part was checking it. A cheaper model can write summaries that read perfectly and still change what search returns, so we graded each model on the search its digests feed, checking whether the same query returned the same traces.

Three panels: a raw agent trace tree on the left, one structured LLM digest call in the middle, and a semantic search over the results on the right.

A messy trace becomes one structured digest call, and the digests are what you search.

Motivation

We build Glassray, an eval platform for AI agents in production. To know whether an agent did its job, you read its traces, so trace data is what we work with all day. The digest is how we make that data usable: our search, clustering, and monitoring all run on it, not on the raw spans. It has to be accurate, and because it runs on every trace, it also has to be cheap.

Results

Every model got the same inputs: a frozen set of 50 trace inputs (real traces plus synthesized ones spanning many languages, code, SQL, multi-tool runs, crashed runs, and oversized traces that trip our truncation caps), three full runs each. Four cheaper models were the candidates; Claude Sonnet 4.6, the model we run today, is the control that sets the bar. To pass, a candidate had to hold Sonnet's per-rule pass rates within a calibrated noise band and keep search returning the same traces. The calibration and the rule-by-rule evidence are below; the one-line summary, cost and reliability on the left and quality against the Sonnet baseline on the right, is here:

Model Cost ($/1k) Latency Valid JSON Quality rules Language Topic Search overlap Verdict
Claude Sonnet 4.6 (today) $3.45 ~3s 99.3% baseline 100% 0.96 0.79-0.84 reference
Claude Haiku 4.5 $1.11 ~2.1s 100% all held 94% 0.83 0.62-0.63 ✅ second closest
gpt-4o-mini $0.11 ~1.6s 100% held 92-96% 0.76 0.51-0.55 ✅ winner
Kimi K2.5 $1.61 ~21s 86.7% 5 regressions 30-47% 0.67-0.76 0.56-0.63 ❌ fails
Gemini 2.5 Flash-Lite $0.06 ~1.0s 92.0% 4 regressions 96-98% 0.71 0.52-0.56 ❌ fails

Reading the columns:

  • Quality rules: how the candidate scored against Sonnet on the seven trace-grounded rules the digest promises, after the plus-or-minus-4-point noise band is applied. "Held" means no rule dropped beyond the band; "N regressions" counts the rules that did.
  • Language and Topic compare the candidate's language tag and topic label against the baseline. Kimi's language score collapses because it writes "English" where the schema wants en.
  • Search overlap is top-3 nearest-neighbor overlap against the incumbent's corpus, same embedding model as production. Sonnet's own 0.79-0.84 (it against a second Sonnet run) is the ceiling any candidate could reach; ranges span the three runs.

How the digest works

A raw agent trace is a large, unstructured set of calls. One user request fans out into dozens of spans (planner calls, tool calls, retries, sub-agents), each carrying prompts, payloads, and framework metadata. Here is one real request:

One real request fanning out into dozens of nested spans, none of it searchable yet.

One request becomes dozens of nested spans that search can't reach.

Dashboards built on raw spans answer "what happened in this run." They can't answer the questions you actually ask when something feels off: which runs are about refunds, or which traces resemble one that broke. For those, each trace has to be reduced to something structured:

  • a summary, two or three factual sentences saying what the user asked and what the system did,
  • a topic, a short label like refund or data deletion,
  • a language tag, what language the conversation was actually in,
  • an embedding, the vector that makes semantic search and clustering work.

The digest is one structured LLM call per trace, plus an embedding of the summary. The call reads only what is already on the trace row (the user request, the step names, and the final answer, each bounded at write time). It never re-fetches the raw payload. That keeps the call small and its cost predictable, whatever the size of the original trace. The design is simple; three choices decide whether it holds up in production.

Make the output structured

Every digest returns a fixed schema (summary, language, topic) with the output language pinned: summaries and topics are always English, the dashboard's language, while the language field reports what the trace was actually in. Without pinning this, a Spanish support conversation gets a Spanish summary, and your clusters split by language rather than topic.

Big traces get cut

Bounded text means truncation, and we bound it rather than try to fix it. At ingest, each stored field (the user request, the final answer) is capped at 8,000 characters, and the assembled digest input at 12,000. The assembly order protects the trace's skeleton: the user request first, then the step names (so the model always sees which agents and tools ran), then the final answer. Because no single field can exceed 8,000 characters, even a maxed-out request leaves room for the steps and the start of the answer. What falls off first is the tail of a very long answer, so the answer itself stays intact, and the cut only affects what the summarizer reads: the stored answer stays on the trace, where search lands you anyway. What we do require is honesty about the cut. The model has to stay factual about the part it can see, and must not invent an ending for text that stops mid-sentence. For judgment-heavy work we run a cheap selection pass that picks the relevant steps of an oversized trace; the digest deliberately keeps the dumb-but-cheap cut instead, because at this volume a second LLM call to choose what to read would cost more than the digest itself. We made the cut an explicit test: our evaluation set includes traces built to overflow both caps, scored by a dedicated rule ("no inventing past the cut").

What the digest model sees on a big trace: the user request (8,000-character ingest cap) and the step names, then the final answer (8,000 characters), then a truncated tail the model never sees, all inside a 12,000-character digest window.

What the digest model sees on a big trace. The tail of a long answer is the first thing to fall off, and the rule that guards it is to stay factual about the visible part and never invent the ending.

Consistency is the hard part

The digest's job is to produce the same kind of summary every run, because everything downstream is built on it (trace to digest to embedding to search and clusters).

The embedding is computed from the summary text, so similar text lands at nearby points in vector space. If the digest describes a trace the same way every time, its embedding lands in the same place, and the same query keeps returning that same trace. The goal is that the same query returns the same results whether you run it today or next week, on this model's digests or a cheaper one's.

This kind of drift raises no error. If a summary shifts in style or focus between runs, its embedding moves with it, so search neighborhoods reshuffle and clusters relabel. The summary still reads fine even though the search results no longer hold. So reading the summaries is not enough. You have to run the search the digest feeds and check that the same query returns the same top traces on every run.

A pipeline flow: Trace, then Digest (summary, topic, language), then Embedding (one vector per trace), then Search and clusters. A marker at the digest reads 'the model swap happens here', and a marker at search reads 'but you validate here'.

The model swap happens at the digest, but its effect only shows up at the thing that consumes it, so that is where you validate.

The experiment: the cheapest model that holds the bar

The digest is the highest-frequency LLM call we make, running once for every trace across every customer. Today it runs on Claude Sonnet 4.6, already a step down from the larger model we use for judgment-heavy work. The question is how much cheaper we can go before search results change.

Where the baseline stands today, measured over 150 metered calls:

  • Cost: $3.45 per 1,000 digests ($0.0035 per trace)
  • Latency: about 3 seconds per digest
  • Reliability: 99.3% valid structured output (149 of 150 calls)
  • Quality: judged against the source trace, and this is the reference. A pinned judge scores every digest, Sonnet's included, on seven trace-grounded rules; Sonnet's pass rates are the baseline each candidate is compared against.

Routing high-volume work to a smaller model is a well-known pattern. Verifying the swap takes four checks: that each digest is still accurate to its source trace, that the model still returns valid structured output on every call, that the consumer (search and clustering) still returns the same results, and a calibration for telling real regressions from run-to-run noise.

The setup

We ran the ladder in Glassray Coach, our local, open-source experiment harness, and the reports in this post are its output, unedited. A frozen set of 50 trace inputs, real traces plus synthesized ones covering 19 languages, code, SQL, multi-tool runs, crashed runs, steps-only traces, and three oversized traces that trip the truncation caps. Five models, three full runs each (750 digests total), all on the production code path with real metered cost per call. Each measurement uses the slice it needs: cost, latency, and reliability are aggregated over all three runs (150 calls per model). The search check (for each trace, do its nearest neighbors in embedding space stay the same as with the incumbent's digests) is reported as a range across the three runs, and the seven fixed quality rules are judged on the first run of each model (250 digests, one pinned judge so scoring never varies). The seven rules are the digest's own promises, lifted from its prompt and schema:

  • summary and topic always in English;
  • never invent details;
  • report the trace's real language;
  • say so plainly when the text is empty;
  • use plain everyday English (the one subjective style rule);
  • no inventing past the cut on truncated traces;
  • the summary is two to three sentences covering what was asked and what happened.

The ladder: four candidates, each measured head-to-head against the incumbent: Claude Haiku 4.5, gpt-4o-mini, Kimi K2.5 (via OpenRouter), and Gemini 2.5 Flash-Lite (via OpenRouter). Each comparison below is one of those match-ups: the candidate's digests of the 50 traces, scored rule by rule against Sonnet 4.6's digests of the same traces.

Why the rules, not just the summary

Reading the summaries and deciding they look fine is the obvious check, and it is the one that misses the most. A cheaper model's digest can read perfectly and still be wrong: it can state a detail the trace never held, tag a Spanish conversation as English, or shift its phrasing just enough to change which traces a search returns. None of that surfaces from the text alone. It surfaces when you hold the output against the rules the flow is supposed to follow.

Those rules are not written by hand for the experiment. Coach reads the digest flow's own code and prompt and pulls out the promises it makes, each one tagged with the file it came from.

The Glassray Coach CLI running 'flows discover' on the digest flow. It reads apps/worker/src/watcher/digest.ts and lists seven rules derived from it, each labeled with its source file: summary and topic in English, never invent details, report the real language, say so when text is empty, use plain everyday English, no inventing past the cut, and the summary is 2-3 sentences covering the ask and outcome, all from digest.ts.

Coach reads the digest flow and lists the rules it derived, each tied to the file it came from.

Each rule is one checkable promise, like "report the trace's real language" or "no inventing past the cut." Coach scores every run against all seven and quotes the evidence behind each pass or miss. So the reports below are not a judgment on whether the summaries look good; they are a per-rule comparison against the model we run today, on the same 50 traces.

Calibrating the noise band

Coach runs the judged side of the experiment (the seven rules, the pinned judge, the per-model comparisons) and meters cost, tokens, and latency on every call as the traces land. The search check is our own script over the stored digests, using the production embedding model. Before judging any candidate, we ran the incumbent against itself in Coach: same model, same 50 inputs, two independent runs, compared exactly like a candidate match-up. That Sonnet-vs-Sonnet run is the control, and its baseline is the first Sonnet run. Individual rules still shifted by a few points between identical runs, and that shift sets the noise band, plus or minus 4 points per rule, which everything below applies to: only drops beyond the band count as real regressions.

A Coach report titled 'Noise floor: Sonnet vs itself', comparing two independent Sonnet runs on the same 50 traces; two rules move by 2 points, and that spread sets the noise band.

The control: Sonnet against itself on the same 50 traces. Its rules still move a few points between identical runs, and that spread is the plus-or-minus-4-point band every candidate is judged against.

Where each model held up, and where it broke

With the band in hand, start with the bottom line: what each model costs against how many of the seven rules it actually broke.

A scatter plot of cost per 1,000 digests (log scale, x-axis) against how many of the seven rules each model broke beyond the noise band (y-axis). Gemini 2.5 Flash-Lite is cheapest but broke 4 rules; Kimi K2.5 broke 5 at a higher price; gpt-4o-mini broke one rule at a thirtieth of Sonnet's cost and is ringed as the pick; Haiku 4.5 broke none; Sonnet 4.6 is the reference.

What each model costs against the rules it broke. gpt-4o-mini holds all but one rule at a thirtieth of Sonnet's price; the cheapest model, Gemini, broke four.

Price is a poor predictor here. The cheapest model on the sheet, Gemini, broke four rules; Kimi broke five at a higher price. Yet gpt-4o-mini holds all but one rule at a thirtieth of Sonnet's cost, and Haiku holds every one. A count of broken rules doesn't say which rule broke or by how much, so drop to the rule level, every delta against the same band:

A strip plot with one column per candidate and one dot per rule (seven per model), plotting each rule's pass-rate delta against Sonnet in points, with a shaded plus-or-minus-4-point noise band around zero. Grey dots sit inside the band, green dots are improvements above it, red dots are regressions below it. Haiku sits tight to the band with one improvement; gpt-4o-mini stays in the band except one deep outlier (style rule, minus 24); Kimi and Gemini drop several rules below the band, the deepest labeled language minus 18 and summary shape minus 20.

Every rule's delta against the ±4-point band. Grey dots sit inside the noise floor; the red dots below it are the real regressions, and the three deepest are labeled.

Behind each table row is the full comparison: for every rule, how often each model kept the digest's promises on the same 50 traces (no invented details, honest about the cut, format held), plus tokens, cost, and latency on each side. The reports flag every negative delta; the table's verdicts apply the plus-or-minus-4-point band, which is why Haiku shows one flagged rule but counts as held.

The per-model specifics:

  • Haiku's one flagged dip ("never invent details," 96% to 92%) is two traces out of 50, the same amount the Sonnet-vs-Sonnet control moved on its own rules, so it stays inside the band; two of its other rules improved.
  • gpt-4o-mini's single real drop is the subjective style rule, 84% to 60%, its summaries reading a little stiffer. Whether that matters for a dashboard label is a product call.
  • Kimi prices cheap per token but writes about 727 output tokens per digest, ten times everyone else, which makes it more expensive than Haiku at 21 seconds a call.
  • Reliability split by hosting: the direct-API models returned valid JSON on effectively every call, while the two OpenRouter models did not (Gemini's invalid rate rose across its three runs, 4%, 6%, then 14%). In production, a failed call is a retry or a missing digest.

First, the two that passed:

Coach verdict card, sonnet-r1 vs haiku-r1: 6 of 7 rules held, one flagged ('never invent details', 96% to 92%, inside the band), cost down from $0.1724 to $0.0556 per 50 traces.

Haiku 4.5 against Sonnet: six of seven rules held, and the one flagged dip sits inside the noise band.

Coach verdict card, sonnet-r1 vs gpt-4o-mini-r1: three rules flagged, but only the plain-English style rule (84% to 60%) falls outside the band; two rules improved; cost down from $0.1724 to under $0.01 per 50 traces.

gpt-4o-mini against Sonnet: two rules improve, two of the three flagged drops are within noise, and only the subjective style rule is a real regression, the model we shipped.

Now the two that failed:

Coach verdict card, sonnet-r1 vs gemini-r1: 0 of 7 rules held, all seven regressed beyond the band.

Gemini 2.5 Flash-Lite: every rule regressed beyond the band, and it also returned unparseable JSON on a growing share of calls.

Coach verdict card, sonnet-r1 vs kimi-r1: one rule held, six regressed; the candidate wrote 14,858 to 31,488 tokens, far more output than the others, at 22 seconds a call.

Kimi K2.5: one rule held, and it wrote roughly ten times the output tokens of the others, which made it slow and, per call, expensive.

Checking that search still works

A cheaper model whose summaries look fine can still degrade search: different phrasing shifts the embeddings and reshuffles the neighbors, with no error to warn you. This check proves that after the model swap, search over the digested corpus returns the same traces as before.

How the check is built. We reuse the digests from the ladder: same 50 traces, one set of summaries per model. Each summary gets embedded with the same embedding model our production search uses. Then, for every trace, we take its 3 closest neighbors in Sonnet's set and in the candidate's set, and check how much the two lists agree; if the neighbors change, search and clustering change too. Even two runs of the same model don't agree perfectly, so instead of comparing against a perfect 1.0, we compare against how well Sonnet agrees with itself (0.79-0.84).

The results. Haiku lands closest to the ceiling (0.62-0.63); gpt-4o-mini sits lower (0.51-0.55) with all its rules held. Its digests phrase things differently than Sonnet's while staying accurate, and the phrasing differences only surface when you search. As an end-to-end spot check, we ran one realistic query ("customer wants a refund or complains about billing") over the incumbent's corpus and the winner's, ranked by cosine similarity: 4 of 5 top results are identical, and the fifth is a relevant billing trace in both.

Semantic search over the digested corpus for the query 'customer wants a refund or complains about billing', showing the top five results side by side: 'before' on Sonnet 4.6 digests and 'after' on gpt-4o-mini digests. Four of the five are the same traces.

The same query over Sonnet's digests and gpt-4o-mini's. Four of the five top results are identical, and the fifth is a relevant billing trace in both.

Limitations

  • Under-specified inputs make judge pass rates look more decisive than they are: on a mixed English/Spanish trace, Sonnet says en, the candidates say es, and the judge passed all of them, because neither answer is wrong.
  • Subjective style rules are judge-sensitive: re-judging identical stored digests after an infrastructure change moved the style rule's verdicts by several points while the objective rules barely moved. Treat style rules as a signal rather than a hard gate.
  • These results are for our digest prompt and trace mix. The ladder and the method should carry over to other setups, though the exact numbers will differ.

What we're changing

A cheaper model can produce the digest well enough, so we are moving the digest to gpt-4o-mini. It held six of seven rules, all but the subjective style one (see limitations), and search still returns the same results. On cost, gpt-4o-mini runs 1,000 digests for $0.11 against Sonnet's $3.45, so at a million traces that is roughly $110 instead of $3,450, a saving of about $3,340, and at half the latency. Haiku 4.5 is the conservative runner-up if we ever want behavior closest to the incumbent.

The two models that looked cheapest on the price sheet both failed, one on quality and reliability, one by writing ten times the tokens. The price sheet doesn't know your workload, so validate on your own hot path (against whatever consumes the output) and calibrate by running your incumbent against itself.

Coach is open source, and there is a quickstart if you want to run this ladder against your own agent's hot path.

If you are building AI agents in production and want a better way to make sense of your traces, we would love to hear from you. Join the waitlist, or find us on X: Glassray, Manuel, Alex.