A ChatGPT answer doesn't stay in the chat window. It gets pasted into a PR description, quoted in a design doc, repeated in a meeting as "apparently…" — and at that point its errors become your errors, with your name attached. Fact-checking the answer before you repeat it is not paranoia; it's the same hygiene as running the tests before you merge.

The good news: you don't need to verify everything, and you don't need a research afternoon. You need a triage rule and one copy-paste fact-check prompt — it works the same in ChatGPT, Claude, or Gemini. Both are below, along with a real test run and an honest account of where the short version runs out of road.

Why fact-check ChatGPT answers at all?

Because the model is optimized to be plausible, and plausible is not the same as true. Language models produce the answer that best fits the shape of the question — which usually overlaps with the truth, and sometimes doesn't. The failure mode is not obvious nonsense; it's the confident sentence that is 90% right with one wrong load-bearing detail. No amount of fluent wording distinguishes the two from the outside. Only checking does.

Which claims are actually worth checking?

Not all of them — that's the mistake that makes people give up on verification entirely. The triage rule I use:

Check the claims that would change your decision if they turned out to be wrong. Skip the rest.

In practice that means three categories get checked, in order:

  1. Claims you're about to act on — "this API is rate-limited at X", "this license allows Y".
  2. Claims you're about to repeat to someone else — anything heading into a doc, a PR, or a conversation where you'll be the source.
  3. Specifics that smell too neat — precise percentages, named studies, tidy version histories.

Everything else — background info, general context, phrasing — can stay unverified, because being wrong there costs you nothing.

The copy-paste ChatGPT fact-check prompt

Once triage hands you a claim that matters, stress-test it. Paste this into a fresh chat — deliberately not the chat that produced the claim, so the model has no stake in defending itself:

Stress-test the claim below. Do not assume it is true.
1. Restate the claim precisely; note any ambiguity.
2. Falsifiability: what evidence would prove it wrong?
3. List the 2-3 conditions the claim silently depends on.
4. Name the most likely confounder or alternative explanation.
5. Verdict: SUPPORTED / UNCLEAR / DOUBTFUL, one line why,
   and the single fastest check a human should run.
Claim: [PASTE ONE CLAIM]

Enter fullscreen mode Exit fullscreen mode

The design is intentionally adversarial. Most "is this true?" prompts invite the model to agree with itself; this one forces it to construct the case against the claim — what would falsify it, what it silently depends on, what else could explain it. Agreement has to survive the attack to show up in the verdict.

What happens on a real claim?

I fed it a claim you've probably heard stated as fact in a code review: "Using an ORM prevents SQL injection." One pass, Claude Sonnet, fresh context — the prompt itself is model-agnostic and pastes into ChatGPT unchanged. What came back:

  • The ambiguity surfaced first: does "prevents" mean immunity or risk reduction? Does "using an ORM" cover the raw-SQL escape hatches every ORM ships?
  • Falsifiability, concretely: one real case of an injectable ORM codebase disproves it — and the pass pointed exactly where such cases live: raw() / text() / execute() escape hatches with string concatenation, and untrusted input used as identifiers (table names, column names, sort direction), which ORMs typically don't parameterize — they parameterize values.
  • The silent conditions: developers never touch raw-query APIs; untrusted input never reaches identifier position; the ORM's own parameterization is bug-free.
  • The confounder — my favorite part: the thing that actually prevents injection is parameterized queries. The ORM just defaults you onto that path. Crediting the ORM confuses the tool with the mechanism.
  • Verdict: DOUBTFUL — risk reduced, not eliminated — with a fastest-check I'd genuinely use: grep the codebase for the ORM's raw/execute escape hatches and look for string concatenation.

That's a better security review of the claim than most humans give it, and it took under a minute. Note what made it work: the claim was load-bearing (people skip input sanitization because of it), and the prompt attacked instead of agreeing.

Where does the one-claim prompt stop?

Used daily, the short version has four walls you'll hit:

  • One claim at a time, and you do the extracting. A full ChatGPT answer contains a dozen claims tangled into prose; pulling them out and picking the load-bearing ones is on you. (That extraction step is a separate tool — see the hallucination checker from part one of this series.)
  • No graded confidence. SUPPORTED / UNCLEAR / DOUBTFUL is three buckets; when you're deciding whether to ship, "how doubtful, exactly?" matters.
  • No evidence structure. The short prompt names what evidence would settle the claim, but doesn't organize what supports versus what refutes — you rebuild that on your desk.
  • Consistency drifts. Freeform output means the depth of the audit varies run to run and model to model.

I stress-test claims often enough that I maintain the full version as a paid prompt: Fact Checker: Claim Stress Test on PromptBase. Same adversarial core, plus a 1–5 confidence rating, structured evidence-for/evidence-against, and the same audit depth on every run — built to push back, not agree.

Hallucination check or fact-check — which one do you need?

They're different passes and they compose:

  • Hallucination check (part one of this series): sweep a whole AI answer, extract every claim, flag fabrication patterns. Broad and shallow — it finds the suspects.
  • Claim stress test (this article): take one claim that matters and pressure-test it in depth. Narrow and deep — it interrogates the suspect.

Sweep first when the whole answer is unvetted; stress-test the survivors you're about to act on. On a claim someone hands you in isolation — a stat in a meeting, a "best practice" in a review — skip the sweep and go straight to the stress test.

The fast AI fact-checking routine

Whatever the verdict, the model's word is the map, not the territory. The closing routine:

  1. DOUBTFUL verdicts: run the "fastest check" the audit named. It's usually a grep, a changelog, or one official doc page.
  2. SUPPORTED verdicts on high-stakes claims: still spot-check the single strongest condition it listed. Support from a model is an argument, not a source.
  3. Anything with a named study or precise number: search the exact source. Can't find it fast? Treat it as fabricated and cut it.
  4. Never repeat a claim more confidently than your verification supports. "Docs say X" and "a model reasoned X" are different sentences — keep them different.

Written with AI assistance; the test run above was performed as described (one pass, Claude Sonnet, fresh context) and reported faithfully in summary. Reproduce it: paste the ORM claim into the checker and compare.

Full version: Fact Checker: Claim Stress Test. Previous in this series: How to Catch AI Hallucinations.