The defect got found twice before it ever shipped. Once by a reviewer, who wrote it up in the pull request. Once more by a pre-release audit, which ranked it the single most important thing on its list.
It shipped anyway. Then it reached production, where somebody found it a third time.
I’d spent a year building the thing that was supposed to prevent exactly that.
What I Expected the Audit to Find
My team built a merge bar: the evidence a change owes before a named human signs it, scaled to blast radius rather than diff size. We piloted it across four brokerage applications, and it did what we hoped — it caught critical defects before they reached UAT.
Later we carried the same rules onto something much smaller and entirely separate — a different application, nothing load-bearing, two of us building it over about three weeks. Eighty-one pull requests, and seventy of seventy-one merges signed by a human.
Then somebody with no stake in how we’d built it audited the finished thing end to end. I thought I knew what the list would look like. Blind spots. Questions nobody thought to ask. The gaps you get when a reviewer is perfectly competent and the process never pointed them at the right thing.
I’d had a rule ready for that, and I liked it: turn every escape into a named review lens, permanently, so that class of defect can’t escape twice. Every production failure teaches the process something it keeps.
The audit came back with nine defects that had reached production.
Not one of them was a simple failure to look.
The Number That Broke My Rule
Seven of the nine were within review’s reach. Three were the kind my rule fixes — nobody was asked the question, so nobody answered it. Add the lens, close the gap, and that one’s genuinely dead.
The largest group was something else. Those defects had been found. Written down. Dated, in a review or an audit report, before the merge. And shipped regardless.
A lens is a form of memory. It assumes the failure was forgetting, and it fixes forgetting beautifully. But you can’t add a lens for “we knew.” The finding was already there, in writing, dated before the merge. Point a fresh reviewer at the same question next time, and you get the same note, and the same note gets you the same outcome, because the note was never the problem.
That’s the part we got wrong, and it took an outsider to show us. We’d built an instrument for attention and assumed it covered disposition. Attention and disposition are different failures, and they need different machinery. What I was missing was a rule for what happens to a finding when someone decides not to act on it — who decides, where the decision gets recorded, and what it takes to reopen it. Four of those nine needed that, or needed its sibling: a trigger that re-checks a fix when the thing around it moves.
Neither of those is a lens. We’d built one instrument, and I’d called it a system.
Two Reviews, Opposite Answers, Nobody Noticed
One of the nine is worth describing on its own, because it’s the failure mode I now look for first.
An access-control review walked the payload field by field. It enumerated what was there, then concluded, in a sentence, that the payload was safe — a conclusion its own enumeration contradicts. Read the list, then read the verdict, and they don’t agree.
That same week, a separate audit flagged the same field independently.
Two review artifacts reached opposite conclusions about the same thing, and the permissive one won. Not because anyone weighed them. Because nothing in the process was watching for the disagreement. We had a rule for dissent inside a review — one reviewer objecting is enough to hold a merge. We had nothing at all for dissent between two reviews. The one that said “fine” was just the one somebody read.
And this gets harder as you add reviewers, not easier. Every independent pass is another chance to produce a contradiction that nothing in the process is looking for.
The Part I Have to Say Out Loud
The audit found the nine. We sorted them into categories. And the boundary that matters most — between “no reviewer was pointed at this question” and “somebody looked right at it and missed it” — is a judgment call made by the people who designed the review process.
We’re not disinterested. A lens gap is an architecture problem. A plain miss is a people problem, and it’s the less flattering of the two. Read “not one was a simple failure to look,” knowing who did the sorting.
What isn’t a judgment call is the group the argument actually rests on. For those, the finding exists in writing, dated before the merge. You can go and look. Anyone can check that count, and it’s the one that broke my rule.
What to Do Monday
If you run a review process you’re proud of, the useful exercise isn’t counting what it catches. It’s getting someone with no stake in it to audit something you’ve already shipped, then sorting what they find by why it got through.
Then build for what you actually see:
- A lens, for the question nobody was asked. This is the instrument most teams already have, and the one most writing is about.
- A disposition rule, for the finding that was raised and dropped. Who can close a finding without a fix, where that decision lives, and what reopens it. A finding with no recorded disposition isn’t closed; it’s lost.
- A re-check trigger, for the fix that decayed when something around it moved.
And one cheap thing that costs nothing: when two reviews of the same change reach different conclusions, make the process notice. Right now most of them don’t, and the permissive answer wins by default.
What I learned: this was one audit of one system, and I wouldn’t build a law out of nine defects. But it pointed at something I hadn’t considered at all — a mature review process may not fail by not looking. It fails at the moment somebody has already looked, written it down, and the team decides — usually without deciding — to ship. That named-lens rule I was so pleased with addressed a third of our escapes. I’d mistaken one instrument for a system, and only an audit I didn’t run could show me.
Every team adopting AI-assisted engineering is about to generate far more findings than before. The bottleneck was never going to stay at finding things. It moves to what you do with a finding once you have it.
So go and look at the last thing your team shipped, and count the findings raised against it before the merge. Then find out what happened to each one. If you can’t tell a considered risk from a dropped one — and most of us can’t, because nobody wrote the decision down — you have just found the thing to build next. It isn’t another reviewer.