The PR Check That Reads Your README

Docs don't rot because people are lazy. They rot because updating them is nobody's job at the only moment it could happen — merge time.

The most consistently stale file in any repo

Every repo has one artifact that's reliably out of date: the README. Code gets reviewed, tests get run, builds get validated — and the file that's supposed to explain all of it gets touched twice a year, usually in a panic before someone important looks at the repo.

I had a better example than most. The README on the repo behind this site was 344 lines long, and every one of them described someone else's project. I bootstrapped the site from an open-source starter template, then replaced the content, the branding, the share cards, the avatar — a deliberate, weeks-long de-templating effort that somehow never once included the README. It still had the original author's sponsor badges at the top.

That's not a special failure. That's the default outcome, because documentation has no forcing function. Tests fail loudly when code breaks them. A README fails silently, one merged PR at a time, and no step in the pipeline ever asks whether the docs still match the code.

A pull request is the natural place to ask. And "does this diff contradict this document?" is a reading-comprehension question — which, as I wrote in the failure-triage post, is exactly the kind of question a language model is good at answering cheaply.

What I built

An advisory PR check, running on this site's own repository. On every pull request to main, a GitHub Actions workflow collects the changed-file list and the diff, pairs them with the current README, and asks a model one narrow question: does this change make the README wrong, incomplete, or misleading?

The answer lands as a single sticky comment on the PR — one of three verdicts:

✅ README unaffected
⚠️ README update suggested — with the section named and replacement wording proposed
❓ Not enough context to judge

Repeat pushes update the same comment instead of stacking new ones. The whole thing is one workflow file and about 150 lines of dependency-free Python — the repo is public, so this time I can link the actual implementation rather than a clean-room sketch: readme-check.yml and readme_check.py.

The design rules carry over directly from the failure-triage step, because they're the same rules:

Advisory only. The check never blocks a merge. A missing API key, a model timeout, an API error — every failure path exits clean. I made the full argument for this in the triage post and won't repeat it; the short version is that the failure mode of a wrong suggestion is thirty seconds of skepticism, and the failure mode of a wrong gate is a team that turns the whole thing off.

Context selection over prompt cleverness. The model gets the changed-file list, the diff capped at 30k characters, and the README capped at 25k. Not the whole repo — the question is about this change, and widening the context just invites the model to editorialize.

Skip aggressively. Blog posts, images, and social drafts can't make the README stale, so PRs that only touch those exit before any API call. So do PRs that edit the README itself — if you're updating the docs, you're already doing the thing the check exists to nag you about.

What its first two verdicts taught me

Here's where it got more interesting than I planned.

The checker's first-ever run was on its own PR — and remember, this repo's README was still 100% upstream template. The verdict came back: ✅ README unaffected.

My first reaction was that it had missed the most obvious problem in the repository. It hadn't. Read strictly, the verdict is correct: the PR added a CI workflow, and no claim in the (entirely wrong) README was made more wrong by that diff. The README's staleness predated the change. The check answered exactly the question I built it to answer — did this diff cause drift? — and the debt I wanted it to flag was outside that question's scope.

That distinction turns out to be the design insight of the whole exercise: a diff-scoped check catches drift, not debt. It's a ratchet, not a renovation. It can hold documentation in place once the documentation is true, but it cannot make documentation true, because on any given PR the existing rot is always someone else's problem — which is precisely the dynamic that rotted the README in the first place.

So the renovation had to be a deliberate act. I rewrote the README by hand — 344 template lines down to about 60 true ones — and while I was in there, deleted a leftover template workflow that had been failing on every single push to main since the site launched. That went up as its own PR, which gave the checker its second live test, and this one showed the tool actually working: it read the new README from the branch, checked it against the workflow deletion, and confirmed nothing documented had been orphaned — the README claimed Vercel-only hosting, so removing the dead GitHub Pages workflow left no stale claim behind. ✅ again, this time with reasoning that verified something instead of merely permitting it.

Two verdicts, both green, and they mean completely different things. The first one defined the tool's limits. The second one demonstrated its job.

What I'd tell you before you build one

Pay down the debt first. The check only earns its keep once the README is worth protecting. If your docs are already wrong, the checker will keep truthfully reporting that your PRs didn't make them wronger — forever. Fix the docs once, by hand, then let the ratchet hold.

Keep it diff-scoped anyway. The tempting fix for the first verdict is to widen the question — "also audit the whole README while you're in there." Resist it. A check that re-litigates the entire document on every PR produces long, repetitive comments that developers learn to scroll past, and scroll-past is death for an advisory tool. Narrow and occasionally silent beats thorough and always noisy.

Advisory first, active later — maybe. The obvious next step is having it push a suggested commit to the PR branch instead of a comment. I'm not there yet, and the triage post explains why I'm slow about it: trust is the product, and trust builds fastest when being wrong is cheap. A wrong comment costs nothing. A wrong commit on someone's branch costs exactly the goodwill an advisory tool spent months earning.

The meta-note, since this is the second post in what's apparently becoming a series: this took an afternoon to build, and most of that speed came from the first tool having already settled the hard questions — advisory-only, fail-silent, capped context, one place for output. The second AI-in-the-pipeline check wasn't a second design exercise. It was a pattern instantiation. That's the strongest argument I know for building the first one carefully.