A custom-grammar bot can be reliable right up until someone types a perfectly reasonable sentence that the grammar author did not anticipate.

The uncomfortable choice is usually framed as either:

  • keep the grammar deterministic but frustratingly narrow, or
  • let an AI model interpret everything and accept less predictable behavior.

That is the wrong boundary. You can keep command execution deterministic while using failed conversations to improve the grammar—and optionally use AI to propose interpretations that a human must confirm.

The key is to treat every misunderstanding as a potential regression test, not as an invitation to make the parser more permissive on the spot.

This tutorial builds that workflow for a small JavaScript bot that understands commands such as:

feed Luna 20g
remind me to feed Miso at 19:30

Enter fullscreen mode Exit fullscreen mode

We will add support for natural variations without allowing an AI-generated guess to execute a command.

The invariant: interpretation and execution are separate

Our bot will produce one of three outcomes:

// Parsed successfully and passed domain validation.
{ status: "accepted", command: { ... }, grammarVersion: "2026-08-02" }

// The grammar did not recognize the utterance.
{ status: "no_match", grammarVersion: "2026-08-02" }

// It matched syntactically but violated a domain rule.
{ status: "invalid", reason: "amount_out_of_range", grammarVersion: "2026-08-02" }

Enter fullscreen mode Exit fullscreen mode

Only accepted commands are eligible for execution. A model suggestion, support reply, or partially parsed command is never equivalent to accepted.

That distinction matters because the user’s real concern is not whether AI can produce a plausible interpretation. It often can. The concern is whether a plausible but wrong interpretation can silently become an action.

Build the smallest reproducible parser

Create the project:

mkdir grammar-repair-loop
cd grammar-repair-loop
npm init -y
npm install peggy zod
npm install --save-dev vitest
mkdir -p src test data

Enter fullscreen mode Exit fullscreen mode

Update package.json:

{
  "type": "module",
  "scripts": {
    "build:grammar": "peggy src/commands.peggy -o src/generated-parser.js --format es",
    "test": "npm run build:grammar && vitest run"
  }
}

Enter fullscreen mode Exit fullscreen mode

Create src/commands.peggy:

{
  function feed(cat, amount) {
    return { intent: "feed", cat, amountGrams: Number(amount) };
  }

  function reminder(cat, time) {
    return { intent: "remind_feed", cat, time };
  }
}

Start
  = _ Please? command:(Reminder / Feed) Punctuation? _ { return command; }

Feed
  = "feed"i __ cat:Name __ amount:Integer _ Unit {
      return feed(cat, amount);
    }

Reminder
  = "remind"i __ "me"i __ "to"i __ "feed"i __ cat:Name __ "at"i __ time:Time {
      return reminder(cat, time);
    }

Please
  = "please"i __

Name
  = QuotedName / BareName

QuotedName
  = '"' chars:[^"]+ '"' { return chars.join(""); }

BareName
  = first:[A-Za-z] rest:[A-Za-z0-9_-]* {
      return first + rest.join("");
    }

Integer
  = digits:[0-9]+ { return digits.join(""); }

Time
  = hour:[0-9][0-9]? ":" minute:[0-9][0-9] {
      return hour.join("") + ":" + minute.join("");
    }

Unit
  = "grams"i / "gram"i / "g"i

Punctuation
  = [.!?]

_  = [ \t\n\r]*
__ = [ \t\n\r]+

Enter fullscreen mode Exit fullscreen mode

The grammar handles polite prefixes, optional punctuation, and quoted multiword names. It still does not decide whether 99:99 is a valid time or whether feeding 5,000 grams is sensible. Those are domain decisions, so they belong outside the grammar.

Create src/parse.js:

import { parse } from "./generated-parser.js";

export const GRAMMAR_VERSION = "2026-08-02";

function validTime(value) {
  const match = /^(\d{1,2}):(\d{2})$/.exec(value);
  if (!match) return false;

  const hour = Number(match[1]);
  const minute = Number(match[2]);
  return hour >= 0 && hour <= 23 && minute >= 0 && minute <= 59;
}

function validate(command) {
  if (command.intent === "feed") {
    if (command.amountGrams < 1 || command.amountGrams > 200) {
      return { ok: false, reason: "amount_out_of_range" };
    }
  }

  if (command.intent === "remind_feed" && !validTime(command.time)) {
    return { ok: false, reason: "invalid_time" };
  }

  return { ok: true };
}

export function parseCommand(rawUtterance) {
  try {
    const command = parse(rawUtterance);
    const validation = validate(command);

    if (!validation.ok) {
      return {
        status: "invalid",
        reason: validation.reason,
        grammarVersion: GRAMMAR_VERSION
      };
    }

    return {
      status: "accepted",
      command,
      grammarVersion: GRAMMAR_VERSION
    };
  } catch {
    return {
      status: "no_match",
      grammarVersion: GRAMMAR_VERSION
    };
  }
}

Enter fullscreen mode Exit fullscreen mode

Do not send parser exception text directly to users. Detailed parser errors are useful to developers, but messages such as “expected Unit at offset 17” are rarely useful support responses and may expose grammar internals.

Make examples the durable record

A chat transcript explains what happened once. A fixture prevents it from happening again.

Create data/grammar-cases.json:

[
  {
    "id": "basic-feed",
    "utterance": "feed Luna 20g",
    "expected": {
      "intent": "feed",
      "cat": "Luna",
      "amountGrams": 20
    }
  },
  {
    "id": "polite-feed",
    "utterance": "Please feed Luna 20 grams.",
    "expected": {
      "intent": "feed",
      "cat": "Luna",
      "amountGrams": 20
    }
  },
  {
    "id": "quoted-name-reminder",
    "utterance": "remind me to feed \"Sir Pounce\" at 07:30",
    "expected": {
      "intent": "remind_feed",
      "cat": "Sir Pounce",
      "time": "07:30"
    }
  }
]

Enter fullscreen mode Exit fullscreen mode

Then add test/grammar.test.js:

import { describe, expect, it } from "vitest";
import cases from "../data/grammar-cases.json" with { type: "json" };
import { parseCommand } from "../src/parse.js";

describe("command grammar", () => {
  for (const fixture of cases) {
    it(fixture.id, () => {
      const result = parseCommand(fixture.utterance);

      expect(result.status).toBe("accepted");
      expect(result.command).toEqual(fixture.expected);
    });
  }

  it("rejects an excessive feeding amount", () => {
    expect(parseCommand("feed Luna 5000g")).toMatchObject({
      status: "invalid",
      reason: "amount_out_of_range"
    });
  });

  it("does not reinterpret unrelated conversation", () => {
    expect(parseCommand("Luna already ate today")).toMatchObject({
      status: "no_match"
    });
  });
});

Enter fullscreen mode Exit fullscreen mode

Run the corpus:

npm test

Enter fullscreen mode Exit fullscreen mode

When a new wording fails, add it to the corpus before changing the grammar. The desired sequence is:

  1. Reproduce the misunderstanding with a fixture.
  2. Confirm the intended command with the user or domain owner.
  3. Run the test and observe it fail.
  4. Make the narrowest grammar change that supports it.
  5. Run the entire corpus, including rejected inputs.
  6. Review whether any old utterance now produces a different command.

That last step is more important than simply reaching a green test suite.

Record a repair case without treating chat as ground truth

A useful repair record needs the original outcome and the confirmed intent. It should not assume that the first developer—or a model—guessed correctly.

const repairCase = {
  id: crypto.randomUUID(),
  rawUtterance: "could you give Luna twenty grams?",
  grammarVersion: "2026-08-02",
  parserOutcome: "no_match",

  // Added only after explicit confirmation.
  confirmedCommand: null,

  source: "in_app_support",
  consentToRetainExample: false,
  status: "needs_clarification",
  createdAt: new Date().toISOString()
};

Enter fullscreen mode Exit fullscreen mode

A practical state sequence is:

needs_clarification
  -> intent_confirmed
  -> fixture_added
  -> grammar_changed
  -> verified
  -> released

Enter fullscreen mode Exit fullscreen mode

Do not automatically retain every failed utterance. Commands can contain names, schedules, account details, or text pasted into the wrong box. Ask for permission before preserving an utterance as a long-lived test fixture, or replace identifying values with representative placeholders.

For example, a confirmed report containing a real pet name could become:

{
  "utterance": "could you give ExampleCat twenty grams?",
  "expected": {
    "intent": "feed",
    "cat": "ExampleCat",
    "amountGrams": 20
  }
}

Enter fullscreen mode Exit fullscreen mode

Where AI helps, and where it must stop

A language model can be useful after no_match because it can propose likely structured interpretations of paraphrases such as “could you give Luna twenty grams?” It can also draft a candidate fixture or group similar failures for maintainers.

That demonstrated capability is not the same as reliable command execution.

A model may:

  • invent an entity that does not exist;
  • choose the wrong intent from an ambiguous sentence;
  • ignore domain limits;
  • produce different answers for the same input;
  • follow instructions embedded in the user’s text;
  • make a confident guess where clarification is required.

Represent model output as a proposal and validate its shape:

import { z } from "zod";

const candidateSchema = z.discriminatedUnion("intent", [
  z.object({
    intent: z.literal("feed"),
    cat: z.string().min(1),
    amountGrams: z.number().int().min(1).max(200)
  }),
  z.object({
    intent: z.literal("remind_feed"),
    cat: z.string().min(1),
    time: z.string().regex(/^\d{1,2}:\d{2}$/)
  })
]);

export function reviewModelCandidate(rawCandidate, knownCats) {
  const parsed = candidateSchema.safeParse(rawCandidate);
  if (!parsed.success) return { usable: false, reason: "invalid_shape" };

  if (!knownCats.includes(parsed.data.cat)) {
    return { usable: false, reason: "unknown_entity" };
  }

  return {
    usable: true,
    proposal: parsed.data,
    requiresUserConfirmation: true
  };
}

Enter fullscreen mode Exit fullscreen mode

Even a usable proposal is not executable. It can power a clarification such as:

Did you mean “feed Luna 20 grams”?

The user’s confirmation can submit a fresh deterministic command. It should not retroactively turn the model response into authorization.

A simple decision framework is:

Situation AI role Human control
Drafting a regression fixture Suggest wording and expected structure Maintainer approves the fixture
Grouping similar failures Recommend clusters Maintainer decides whether one grammar rule covers them
Read-only lookup Suggest an interpretation User confirms if ambiguity affects the result
Scheduling or changing state Clarification only Confirmed command passes deterministic validation
Billing, security, deletion, or irreversible action Do not infer authorization Use explicit, authenticated controls

The useful boundary is based on consequence, not on how fluent the model sounds.

Review semantic diffs before release

Grammar changes are code changes, but their most important diff may not appear in the grammar file. A reordered PEG alternative can change how existing inputs are interpreted while all new tests pass.

Before release, compare the previous and proposed parsers across the approved corpus. Flag these transitions for human review:

no_match  -> accepted   expected expansion
accepted -> no_match    likely regression
accepted -> accepted    inspect if the command AST changed
invalid  -> accepted    verify the domain rule intentionally changed

Enter fullscreen mode Exit fullscreen mode

For every accepted -> accepted transition, compare the complete command object rather than only the intent name. Changing amountGrams from 20 to 200 is a semantic regression even though the intent remains feed.

Keep the grammar version in support records so maintainers can reproduce behavior from the time of the report. A fixture without its parser version can become impossible to diagnose after several releases.

Failure drills worth running

The fixture encodes the developer’s assumption

A user writes “feed Luna at seven.” A maintainer assumes seven grams, while the user meant seven o’clock.

Do not add a fixture until the intended meaning is confirmed. Some utterances should remain ambiguous and trigger clarification permanently.

A broader rule steals inputs from a narrower rule

PEG parsers use ordered choices. Adding a permissive rule above a specific rule can silently change which branch wins.

Include fixtures for overlapping forms, and review semantic diffs whenever alternatives are reordered.

The model suggestion becomes an action through UI wording

A button labeled Continue may conceal what will happen. Display the proposed command in domain language:

Feed Luna 20 grams now

Enter fullscreen mode Exit fullscreen mode

Require a separate confirmation event, then submit the confirmed structured command through the normal validation path.

Rejected examples disappear from the suite

A corpus containing only valid commands rewards increasingly permissive parsing. Keep negative fixtures for unrelated conversation, malformed times, excessive values, and unsupported actions.

A support report cannot be reproduced

If the report omits the grammar version, original parser outcome, or exact approved wording, a maintainer may test against different behavior. Capture those fields automatically while allowing the user to edit or decline the retained example.

Choose a conversation transport after defining the repair loop

The workflow does not depend on a particular support tool. A GitHub issue template, an email alias, or an in-app chat can all collect a failed utterance and ask the user to confirm the intended command.

For a small product, direct founder-led chat can shorten clarification because the person changing the grammar can ask a precise follow-up. The trade-off is that chat is conversational transport, not your regression corpus or release record. Confirmed cases still need to move into version-controlled fixtures.

One implementation option is Knocket, which provides a shareable contact page, an embeddable web live-chat widget, a mobile WebView SDK, and a unified inbox. Its website widget can be installed with a script tag without building a custom backend. Visitors do not need an account to begin a chat, and messages can be routed to Telegram; a quoted Telegram reply can be delivered back to the website visitor.

That can provide the clarification surface, but the parser result, consent decision, confirmed intent, fixture, and release verification should remain in systems you can test and audit.

Release checklist

Before shipping a grammar repair, verify:

  • [ ] The original failure has an approved, privacy-reviewed fixture.
  • [ ] Ambiguous wording was clarified rather than guessed.
  • [ ] The fixture failed before the grammar change.
  • [ ] Existing accepted commands still produce the same structured output.
  • [ ] Negative fixtures remain rejected or invalid.
  • [ ] Domain validation still runs outside the grammar.
  • [ ] AI-generated candidates cannot invoke execution directly.
  • [ ] Stateful actions display an explicit confirmation.
  • [ ] Repair records include the grammar version.
  • [ ] The support response tells the user what changed—or that clarification will remain necessary.

A grammar bug is not fully fixed when one sentence starts parsing. It is fixed when the intended meaning is documented, the old behavior is reproducible, unrelated inputs remain safe, and future changes cannot silently undo the repair.

Disclosure: I work on Knocket, so treat it as one implementation example rather than a neutral recommendation.