Kmemo 2.0 is out, and the two gaps I admitted to in the first post are closed

This is a follow-up to Kmemo: a semantic cache for LLM calls that refuses to serve you the wrong answer. If you have not read it, the one-line version is that Convert 100 USD to EUR and Convert 250 USD to EUR sit at 0.99 cosine similarity, no threshold separates them, and Kmemo reads the candidates as text instead of trusting the number.

That post shipped 1.0 and ended with two things I could not report. 2.0.0 is out now, and both of them have an answer. One of the answers is not the one I was hoping for.

Gap one: I never said what a verifier catches

The figures in the first post were 67% of near misses rejected and 88% of paraphrases kept, and I labelled them guard-only. The lexical chain runs for free; the optional Verifier is a model call that sees whatever the guards let through. I described that residual as the verifier's job without ever measuring how much of it a verifier actually stops.

Against a named reference implementation, sentence_transformers.CrossEncoder over cross-encoder/quora-distilroberta-base:

Four fifths of what the guards miss, and a third of what they had kept. The second column is why this belongs in a table rather than a sentence, and why the verifier stays a seam you opt into rather than a default. Your verifier is not this one; what the table shows is how much of the residual is reachable at all by a model that reads the two prompts.

Gap two: the comparison was against a baseline I built myself

The first post compared Kmemo to a threshold-only cache. That is the thing every "add a semantic cache" tutorial builds, so it is a fair thing to beat, but I wrote it, which makes it a weak claim.

2.0 compares against GPTCache on the same blind corpora, same pairs, retrieval factored out so both sides are handed the identical candidate and asked only whether to serve it.

At zero cost my free lexical chain loses the headline number. GPTCache's ONNX cross-encoder serves fewer false positives than standard(): 0.221 against 0.291 on one split, 0.108 against 0.324 on the other. It gets there by refusing more than half the genuine rephrasings it is shown, where my chain keeps 88%, so roughly half the cache savings is what that strictness costs.

At comparable cost the picture reverses. Their cross-encoder is a transformer inference per candidate, so the row that costs what theirs costs is my guards plus a verifier, which also spends a model call:

Both columns are in the README, including the one I lose. A benchmark reported by the metric that flatters its author does not get believed on any of the others.

One thing worth knowing if you run that comparison yourself. OnnxModelEvaluation.evaluation wraps its entire body in except Exception: return 0, so every internal failure comes back as a similarity of zero and looks identical to a confident refusal. A harness that trusted the documented entry point would have recorded a false-hit rate of 0.000 for GPTCache, which reads like a broken competitor and would have been entirely my own bug. The harness in the repository proves the evaluator works before it believes a single score, and the gate is mechanical rather than semantic: it asks whether the evaluator functions, never whether it agrees with me about a hard pair.

The comments found something I had missed

Someone read the first post and pointed out that every guard receives two strings, and both of them are prompts. The stored answer is right there on the entry and nothing in the match path reads it.

They were right, and it is the one shape the whole chain is blind to by construction:

what is the capital gains tax rate when i sell a second home
what is the capital gains tax rate when i sell a primary residence

Enter fullscreen mode Exit fullscreen mode

No number differs. No unit, no negation, no flipped comparison. Nothing either prompt says is evidence, so every guard abstains, correctly. The cached answer opens Gain on a second home is taxable in full, and it goes to somebody asking about a different house.

2.0 adds a guard that reads it. When the two prompts differ only by a substitution and the cached answer names the word the query replaced, that answer was written for the other question.

SemanticCache(embedder, guards = MatchGuards.responseAware())

Enter fullscreen mode Exit fullscreen mode

It refuses 14 of the 116 near-miss lookups the default chain still serves and none of the 164 rephrasing lookups, and it is still opt-in. The reason is the evidence rather than the result: those answers were written by me for this measurement, because no corpus of real paired answers exists to harvest. A semantic cache corpus records prompts, and the near misses worth catching are exactly the ones whose prompts look alike. That makes it a regression check rather than the blind measurement every other guard is held to, and folding the two together under one default would quietly downgrade the evidence behind all of them.

What else shipped

The core is no longer JVM-only. kmemo-core and InMemoryStore compile for the JVM, iOS, macOS, Linux, Windows, JS and WasmJS, at the cost of no new dependency, since kotlin.time.Instant and kotlin.time.Clock went stable in Kotlin 2.4. The Redis, Postgres and Spring adapters stay JVM-only, because they wrap drivers that exist nowhere else.

The interesting part of that port was not java.time. It was the access-ordered LinkedHashMap, which does not exist outside the JVM and which the store's eviction, the exact-match layer and the verifier's memo were all built on.

The default chain has an eleventh guard, for the near miss where nothing changed and something was added. How do I deploy a Rails app against the same question on Heroku has perfect word overlap, so no lexical check saw it. Measured at zero false rejections across all three corpora.

And four opt-in pieces sit around the match path: reranking so each candidate tried adds something the last did not, quantized retrieval that decides which candidates are looked at and never whether one is served, deduplication on write, and adaptive per-scope thresholds that refuse to run without a verifier watching what comes through.

Upgrading from 1.x

Five named breaks, each with who it affects and the edit that resolves it, in the migration guide. The one easiest to miss: Maven users need kmemo-core-jvm, since Maven does not read Gradle module metadata. Gradle builds change only the version number.

implementation("io.github.nacode-studios:kmemo-core:2.0.0")

Enter fullscreen mode Exit fullscreen mode

Apache 2.0, on Maven Central. Repository: NaCode-Studios/Kmemo.

If something here is wrong, or a configuration is missing from the comparison, say so. That is how the guard in the third section got written.