Your RAG copilot can't count — stop letting it try

Your RAG copilot can't count — stop letting it try

A user asked our document-search copilot a completely reasonable question: how many documents a specific person authored. The copilot answered with a number. Confidently. With a nice sentence around it.

The real answer, computed by the database, was 84. The copilot said — because it did what every RAG system does when you ask it an aggregation question: it counted what it could see.

Here's the claim I want to defend, because I think a lot of us are shipping this bug right now:

💡 An LLM should never compute an aggregate. Not counts, not sums, not "most recent". If the question is arithmetic over the corpus, the answer comes from the system of record — the model's only job is to phrase it.

🔢 Why the model's count was doomed before it started

This is a multi-agent document-search copilot for a quality-management product, built on LangGraph and Bedrock. When a metadata question comes in ("documents authored by X"), a search agent asks the documents service for matching rows, reranks and permission-gates them, collapses multiple versions of the same document into one card, and surfaces at most 30 results.

Every one of those steps is correct for search. Every one of them destroys the count:

Stage	What it does	What "how many" means here
Documents service (SQL)	runs the filter against the database	true total: 84
Retrieval	returns the top-k rows	capped
View-permission gate	drops rows this user can't see	fewer
Version collapse	5 versions of one doc → 1 card	fewer still
Surfaced set	at most 30 cards	≤ 30, always

The model sits at the bottom of that funnel. When it "counts", it counts the last row of the table. The user asked about the first row. Everything in between is retrieval hygiene that silently turned a database question into a context-window question.

And this is the part that makes it worse than a crash: the answer is plausible. A count that's capped at 30 and deduped looks like a real number. Nobody squints at "you have 27 documents" the way they'd squint at a stack trace. In a quality-management context, someone might put that number in a report.

🪤 The twist: we had the right number the whole time — and overwrote it

This is the bit that stung. The documents service returns the true total with every filtered query. The search node even stored it in agent state, in a field called total_available_results.

Then, a few nodes downstream, the permission-gate node overwrote that same field with the surfaced count — deliberately, and for a good reason:

return {
    "sources": diversified,
    # Honest "documents found": distinct logical documents after the view gate and
    # version-collapse, so the headline count matches the cards (5 versions → 4 docs),
    # not the service's version-row total.
    "total_available_results": len(diversified_all),
    ...
}

Enter fullscreen mode Exit fullscreen mode

That comment is right! If the UI says "found 12" and shows 12 cards, the numbers should agree. The bug wasn't either write in isolation — it was one state field carrying two different meanings at two different points in the pipeline. Upstream it meant "matches in the database"; downstream it meant "cards on screen". Whoever read it last got whichever meaning was freshest.

💡 A state field whose meaning mutates mid-pipeline is a bug factory. If a value is authoritative, give it its own field and never let anything downstream touch it.

🔧 The fix: the database counts, the model narrates

So that's exactly what I did — preserved the service's true total in its own state field, threaded it to the two places that answer the user, and left the surfaced count alone for the card list and pager.

First, the state gets an untouchable field:

class DocumentsSearchState(OrchestratorState, total=False):
    ...
    total_available_results: int   # surfaced: view-gated, version-collapsed, capped at 30

    # The documents service's true match total for the filter, before the 30-cap and
    # version-collapse. Authoritative for a "how many" count; 0 on a content-only
    # search, which has no exact total.
    total_matching_records: int

Enter fullscreen mode Exit fullscreen mode

Then the outcome node — the one that turns search state into both the user-facing status line and the message the agent reasons over — uses the authoritative number in both:

total_matching = state.get("total_matching_records") or 0
headline_total = total_matching if total_matching > total else total

found_msg = (
    f"Showing {len(sources)} of {headline_total} matching documents"
    if headline_total > len(sources)
    else f"Found {len(sources)} matching documents"
)
...
count_note = (
    f" {total_matching} documents match in total "
    "(authoritative count; use this for a count question)."
    if total_matching > len(sources)
    else ""
)
return f"Returned {len(sources)} document cards to the user.{count_note} ..."

Enter fullscreen mode Exit fullscreen mode

And the agent's prompt closes the loop with an explicit instruction:

For a "how many" question, answer with the authoritative total the tool
reports ("N documents match in total"), never the number of cards shown,
which is capped.

Enter fullscreen mode Exit fullscreen mode

Notice the belt-and-braces structure, because it matters: the number goes into the agent's context (so the model can phrase the answer around it) and into the user-facing headline ("Showing 5 of 84 matching documents"). Even on a turn where the model ignores the note and free-styles a count, the UI is showing the true total right next to it. The prompt asks; the headline doesn't have to.

Three small tests pin the behavior: the count note appears when there are more matches than cards, disappears when everything is shown, and never appears for a content-only semantic search — because a similarity ranking has no crisp "matches" set, so pretending it has an exact total would be the same lie in a new costume.

📅 The sibling bug: "the last 10 days" of what?

Same release, same root cause wearing a different hat. Users asked for "documents created in the last 10 days" and the copilot had to turn that into a date filter — without knowing what day it is. A model doesn't know today's date. It will happily guess one from its training data, which is exactly as wrong as counting its context window.

The fix is a tiny LangGraph middleware that injects the current date into the system message on every model call:

def current_date_context(now: datetime | None = None) -> str:
    day = (now or datetime.now(UTC)).date()
    return (
        f"Today's date is {day.isoformat()} (UTC). Resolve any relative date "
        '("today", "yesterday", "the last 10 days", "since March") '
        "to concrete ISO-8601 dates relative to it before building a date filter."
    )


class CurrentDateMiddleware(AgentMiddleware):
    async def awrap_model_call(self, request, handler):
        base = request.system_message
        line = current_date_context()
        merged = line if base is None else f"{base.content}\n\n{line}"
        return await handler(request.override(system_message=SystemMessage(content=merged)))

Enter fullscreen mode Exit fullscreen mode

One non-obvious detail: it's injected per call and never appended to the persisted conversation. Write "today is 2026-07-30" into the thread history and a conversation resumed next week carries a stale date the model will trust completely.

Both fixes are the same principle: the model knows nothing you don't hand it, and it will answer anyway. Counts, dates, totals — anything that's a fact about your system rather than about language has to arrive as context, computed by something that can't hallucinate.

🤔 Where I'd push back on myself

Honest limits of what shipped:

• Nothing enforces that the model uses the number. The prompt instructs, the tool message supplies, the headline backstops — but there's no output check asserting the count in the model's prose equals the authoritative one. That's an eval I'd like to have and don't yet.
• Content-only searches still have no real count. A semantic ranking over chunks genuinely doesn't have an exact match total, so those turns fall back to the surfaced count. Arguably the honest move there is to refuse to give a number at all. I didn't go that far.
• This only covers counts. Sums, averages, "which author has the most" — any of those would need real aggregation endpoints on the service side. The pattern extends; the implementation doesn't yet.

And the design choice I keep turning over: I handed the model the number inline, as prose in a tool result. The agentic-purist alternative is a dedicated count_documents tool the model calls when it detects an aggregation question — cleaner separation, one more round trip, and a new failure mode where the model doesn't call it. I went with inline because the search already paid for the count; a tool call felt like latency for ideology.

So: where do you draw that line — hand the model the number, or hand it a tool to fetch the number? If your copilot answers "how many" questions today, I'd genuinely like to know which side you picked and whether it's held up.

---

Thanks for making it to the end 🙏 If your assistant has ever announced a total that made you go "wait, that can't be right", I'd love to swap notes on how you fixed it — come find me on LinkedIn.

---