The solution is a fan-out/fan-in pattern with query generation as the shared prerequisite. Once the query is ready, all four independent steps are dispatched concurrently and their results are merged at a barrier before the final response is assembled. User-visible latency shifts from the sum of the four branches to the maximum of them: roughly three seconds instead of 12 in this example.

Implementing this requires a templated prompt structure where each context section is a named, optional slot, and a lightweight routing step that decides which slots to fill before the main LLM call. That routing step is cheap. It can be a similarity lookup against a slot-to-content index, a rule based on the classified intent or a small model call. The investment pays back on every subsequent LLM call in the graph, which sees a cleaner, shorter and more focused prompt.

What this may look like in practice

Define your prompt as a template with named sections: {{system}}, {{tools}}, {{retrieved_context}}, {{history}}, {{user_message}}. At request time, populate only the sections the query actually needs. Empty slots are omitted entirely, not filled with placeholder text.

Prompt structure and KV cache utilization

Once you've reduced the amount of context, the next optimization is arranging it to maximize prompt cache hits. The two concerns are complementary: get the content right first, then arrange it to maximize cache hits.

Most LLM providers maintain a key-value cache of attention computations across requests. When the prefix of a prompt matches a previously processed sequence exactly, the provider skips recomputing attention for that prefix and charges reduced or zero input tokens for the cached portion. This is a free latency and cost win, but only if you structure your prompts to exploit it.

The rule is straightforward: everything stable goes at the top, everything dynamic goes at the bottom. System instructions, tool definitions and fixed few-shot examples form a long, invariant prefix. The dynamically selected context and the user message slot in at the very end. When the next request arrives, the provider finds the cached prefix, skips it and only processes the short dynamic tail.

The connection to  context engineering is direct: dynamic context selection produces a shorter, cleaner dynamic tail, which means the cached prefix covers a larger fraction of the total prompt. The two optimizations compound. A statically ordered, fat prompt with everything included gets zero cache benefit on the dynamic portion and pays full token cost. A dynamically selected, ordered prompt gets the cache hit on the stable prefix and pays only for a small, focused tail.

One practical constraint: the moment you shuffle the order of any static block between requests, the prefix no longer matches and you forfeit the cache hit entirely. This means few-shot examples must be fixed and front-loaded, not dynamically reordered per query. If you need query-adaptive examples, place them in the dynamic section after the static prefix rather than mixing them into the static block. Most providers also require an explicit opt-in flag (Anthropic's cache_control breakpoint, OpenAI's prompt caching on supported models). Check that it’s active in your LLM gateway config before assuming the cache is doing anything.

Combined rule

First: select only what the query needs ( full context). Then: order the prompt as system → tools → fixed examples → selected context → user message. The static prefix gets cached. The dynamic tail stays small. Both savings stack.

Semantic similarity vs. LLM classification

Intent classification and routing are everywhere in agentic systems. For stable label spaces, semantic similarity often outperforms LLM-based classification on both cost and latency.