Most SEO audit tools still ask the questions search engines cared about ten years ago: is your title tag the right length, do your images have alt text, is your sitemap valid. Those still matter. But a growing chunk of how people find content now goes through an AI assistant instead of a search results page, and almost no audit tool checks whether your site is even reachable by one.

I picked up an open issue on open-seo (an open-source Semrush/Ahrefs alternative) to close that gap, and it turned into a good lesson in scoping a feature for someone else's codebase instead of your own.

The actual question

"Is this site AI-readable?" breaks down into concrete, checkable things:

  1. Does robots.txt block the crawlers that build AI search indexes and answer live user requests?
  2. Does the site have an llms.txt — the emerging convention that gives an AI assistant a clean map of your important pages?
  3. Does the site advertise a Markdown alternate for its pages, so an assistant can read clean content instead of parsing full HTML?

Straightforward to state. The interesting part was not flagging noise.

Lesson 1: not every robots.txt block is a signal

A site with Disallow: / for every crawler made a deliberate, site-wide choice. Reporting "GPTBot is blocked!" on a site that blocks everything isn't useful information — it's noise dressed up as an insight. So the check only flags an AI agent when it's treated worse than the generic rules:

const rootUrl = `${origin}/`;
if ((robots.isAllowed(rootUrl, GENERIC_PROBE_AGENT) ?? true) === false) {
  return []; // whole site is closed — not an AI-specific signal
}

Enter fullscreen mode Exit fullscreen mode

I also grouped agents by why they visit instead of firing one issue per bot. GPTBot and ClaudeBot are training crawlers; blocking them is often an intentional content policy and barely worth a heads-up. ChatGPT-User and Claude-User fetch a page live because an actual human asked an assistant about it right now — blocking those is a much bigger deal, because you're breaking a request in progress, not opting out of a training set. Same signal, completely different severity, so they're separate issue types with separate default severities instead of one generic "AI crawler blocked" bucket.

Lesson 2: absence isn't always an error

llms.txt and Markdown alternates are both new enough that almost nobody has them yet. If the check flagged every single audited site for "missing Markdown alternates," it would train users to ignore the audit tool's warnings entirely — the boy-who-cried-wolf failure mode of any linter. Both checks are info severity with copy that says, explicitly, "this is common, not an error." The one genuinely broken state — a site that serves llms.txt but violates its one hard structural requirement (starts with an H1) — is the only case that gets bumped to warning.

Lesson 3: match the codebase's own constraints, not your own instincts

My first instinct for "does this page have a Markdown alternate" was to persist a new boolean column on the crawled-pages table. Then I actually looked at how the project's crawl pipeline works: full page data lives in a database table, but only a slim summary (title, status code, a few other fields) survives between crawl phases as durable workflow state — link lists and everything else deliberately get dropped to keep memory bounded on 10,000-page crawls.

Adding a persisted column meant a schema migration, which meant touching both the SQLite and Postgres schema paths this project supports side by side — a much bigger, riskier diff for a feature that doesn't actually need to survive past the audit run. So instead the flag rides through the same transient summary object the pipeline already carries between phases, and the finding — not the raw per-page flag — is the only thing that gets persisted, as a single site-level issue row. Same pattern the project already used for the file-based llms.txt check; I just extended it instead of inventing a new one.

Lesson 4: extend the open PR, don't fragment the review

I'd already opened a PR for the robots.txt and llms.txt checks. Before anyone had reviewed it, I built the Markdown-alternates check too — and instead of opening a third PR, I pushed a second commit onto the same branch. The maintainer hasn't looked at any of it yet, so there was no in-flight review to disrupt, and one PR that closes three-quarters of the original issue is easier to review than three PRs that each close a quarter of it and reference each other.

Result

19 unit tests, no schema changes, no new external API calls beyond one extra fetch for llms.txt. The PR's open here if you want to see the actual diff: every-app/open-seo#122.

OpenSEO

Open source alternative to Semrush and Ahrefs

OpenSEO is an SEO tool for the people. If tools like Semrush or Ahrefs are too expensive or bloated, OpenSEO is a pay-as-you-go alternative that you actually control.

All-in-one SEO tool for you and your AI agent.

Connect with any agent like Claude Code, OpenClaw or Hermes. We have pre-built skills, but you can build your own to tailor OpenSEO to your needs.

Image

Hosted Version

Try OpenSEO for free on our website. If you want to support the project, a hosted subscription is $10/month.

openseo.so

Why use OpenSEO?

  • Best in class MCP and AI Skills.
  • Modern, simple UI.
    • Focused workflows instead of a bloated, complex SEO suite.
  • No subscriptions.
    • Bring your own DataForSEO API key and pay only for what you use.
  • Fork and vibe code your own custom tool.

Main SEO Workflows

  • Keyword research
  • Rank tracking
  • Competitor Insights
  • Backlinks
  • Site…



The code itself was the easy part. The actual work was reading how someone else already solved similar problems in their codebase before writing a single line — the "how do I avoid noise" and "how do I fit this project's existing shape" questions mattered more than the crawler-detection logic itself.