In a traditional SDLC, validation was someone else's job and it came later. QA teams checked that the software did what it was supposed to. The behavior was repeatable, so eventually you automated the checks and moved on.

So when building Slooster, I did the hard parts first:

  • Building the prompts, tuning them, defining schemas for parameters and outputs.
  • Building a system to pick the provider, the model, the temperature, and other tuning configs per prompt.

Everything worked great locally, so I told myself the validation layer could wait.

Unfortunately, the traditional SDLC model breaks the moment an LLM enters the equation. Remember, LLMs are producing output based on guessing off the context. They're getting better at it, no doubt, but they are still guessing. The same prompt will give you a different answer next time, or even an incorrect one, with nothing changed, so validation must be part of the software itself.

When a model is in the loop, you have to, from day one, build a system that can absorb the unpredictability without a hiccup.

The (un)predictability factor

After deploying Slooster Guide to staging, I gave it a once-over before demoing to my co-founders.

The exact same prompts (and model, and configs) that had been returning me real, criteria-matched vendors for weeks came back with what looked like placeholder data, literally outputting the words Vendor A, Vendor B, Vendor C. Regenerating then brought Vendor D into the mix.

A model that gives you clean output locally for weeks can still hand you garbage on a real call, and no amount of testing beforehand would be able to prevent it. That is the whole case for validating inline, on every actual output, instead of once before you ship.

The layer I built next

I added the validation layer, with retries built in: bad output gets caught and runs the call again. But retries cost time, and the app needs to stay responsive while a user is waiting on it.

So validation isn't free, and it isn't one-size-fits-all. Where it runs, how hard it retries, and whether a human sees the result all depend on your use case.

Letting a model check a model

The deterministic checks come first: a schema validator rejects anything structurally off-spec. But a placeholder like "Vendor A" is structurally valid. It passes a schema check, yet it's still not a valid output. That is a semantic miss, not a structural one, and you need a separate type of check for it.

So the last gate is a model checking a model: a validation prompt judging the output against the same criteria the first prompt was given.

One AI-gate prompt, roughly:

You are checking another model's answer before a user sees it.
The task it was given: [the original request and selection criteria].
Its answer: [the response to check].
Reply as JSON: 'pass' (true/false), 'reason' (one line), 'confidence' (0 to 1).
Fail it if the answer has placeholder names, generic filler, items that don't meet the stated criteria, or anything that reads like example data instead of a real result.

Enter fullscreen mode Exit fullscreen mode

The gate can run on the same model that produced the answer, or on a different provider and model entirely for an independent second opinion. Make it a config choice, so that you don't need a code change to tune.

The three important takeaways:

  • Not every response should take the retry hit, but not every response should be trusted silently either.
  • Surface the confidence you have in an answer. Let the user see it.
  • Depending on the scenario, show the output with a clear way for the user to push back, and loop that feedback straight back into the system so the next attempt knows the bar it missed.

A model shouldn't get the last word on its own output. Keeping a human in the loop is a feature, not a gap you'll close later.

Same question, different answer

A model won't give you the same response on the next call, which makes naive caching a trap, and no caching expensive.

The answer is caching intelligently:

  • Reuse a good result when it makes sense.
  • Give the user explicit control to regenerate when they want it refreshed.

Intelligent caching also pays for validations. Structuring prompts so the stable part is cached offsets a good chunk of the extra cost the validation layer incurs.

Not every prompt is equal

By treating all prompts the same, you'll end up with increased costs and an underwhelming performance.

Instead, match each prompt type to the proper provider and model:

  • A cheap, fast model for the simple calls.
  • A stronger model where the answer actually matters.
  • A suitably selected model for the validation gate.

Per-prompt selection isn't a premature optimization. It's how you keep the whole system responsive while still validating the parts that count.

The cornerstone

I had planned to add validation after the MVP. That staging shock pulled it all the way forward, and I'm glad it did, as I otherwise very likely would not have accounted for getting placeholder data in the response.

The validation layer is an integral part of the system, along with transformers, schemas and specs, and it's the part I'd start with for the next system I build.

When a model is doing the work, validation can't simply be a layer you choose to add. It needs to be the cornerstone you build on.

Again, there is no one-size-fits-all solution here. AI is giving you leverage by doing a lot of the work, but you still need to be in charge, and organizing your thoughts up front matters more than ever.

If you're building software with AI workflows as part of the solution and wrestling with the same unpredictability, I'd be happy to share my experiences and learn about yours.