Shipping AI code you don't understand is a management problem before it's a quality problem. Addy Osmani calls the result comprehension debt: the gap between the code in production and the code any person on the team understands. The decision for you is who must understand a merge, and how you check that they do. More on running teams through this shift is in our engineering management guides for CTOs and VPs of Engineering.
Where does it show up? Three trigger moments
The first is an incident. Production breaks, the person who merged the change opens the same assistant that wrote it, and the debugging session becomes a conversation with a tool that has no memory of why it chose what it chose. If you're tightening that side of the process, our incident postmortem process is where "nobody could explain the change" belongs as a finding.
The second is a new hire. Someone asks for a walkthrough of a module, and the answer is a shrug, because the author merged it without ever holding it in their head. We cover the human version of this problem in the 30-day knowledge transfer plan. The AI version is worse, because nobody ever knew.
The third is review overload. In a long Hacker News thread from October 2026 ("Ask HN: How to deal with shipping AI-written code you don't understand?"), one commenter describes approving pull requests they can't review at that volume. It's a named discussion, not a statistic.
What is the actual risk? What the evidence says
Start with the one controlled study. Anthropic's Alignment team published a randomised trial on 29 January 2026. 52 mostly junior engineers wrote two features using Trio, a Python async library they were learning. One group had an AI sidebar and the other coded by hand. Both groups then took a quiz.
The AI group scored 50% and the hand-coding group 67%, a 17-point gap (p=0.01). The biggest gap was on debugging questions. The AI group finished about two minutes faster, and that difference wasn't significant (p=0.391).
The limits are the authors' own: small sample, short-term, a chat assistant rather than an agent, and preliminary results. Treat it as a warning about what happens when people delegate and skip the thinking, not as a measure of your team. Anthropic also found that how people used the tool mattered. Delegation-heavy patterns scored lowest (24-39%) and comprehension-seeking ones highest (65-86%), in groups of 2-7 people, with no causal claim.
Osmani's essay (14 March 2026) adds the argument, which is his, not data: the assumption that reviewed code is understood code "no longer holds", and code engineers approved without fully understanding "now carries implicit endorsement". Passing tests are necessary but not sufficient. His point that dashboards can look healthy while understanding drains out is the one to take seriously.
Our judgment: no study measures this inside production teams. The rules below are a reasonable response to thin evidence, not a proven fix.
How to roll out an ownership rule and a review gate
Do these in order. Each takes days, not quarters.
1. Name one owner who can explain the merge
Every change has one author of record, and that person answers for it, not the tool. Our AI acceptable use policy template already recommends a named human author of record who answers for every change. This rule adds what that human must be able to do: explain what the change does, what breaks it and how to undo it.
Make it enforceable rather than aspirational. GitHub's CODEOWNERS file (in .github/, the repository root or docs/) auto-requests the listed owners when a non-draft pull request touches files they own. Turn on "Require review from Code Owners" in branch protection and a change can't merge without an owner's review. One owner's approval satisfies the requirement, so list the people who really understand each area, not whole teams.
2. Add an explain-it gate to review
Put three questions in the pull request template: what does this change do, what would break it, and how would you roll it back. The author answers in their own words, without pasting the prompt or the assistant's summary. A reviewer who can't tell the answer was understood asks a follow-up in the thread.
Practitioners on a smaller Ask HN thread from about five months earlier proposed random walkthroughs of merged code, "explain this" questions in review, and putting authors on call for what they ship. These are practitioner suggestions, not tested practice, and we'd start with the template alone. It adds little to a review, and the answers show you quickly who is merging blind.
3. Separate who writes from who checks, and test the tests
One commenter on the first thread has a second AI review the PR for its different biases; another values a different human pair of eyes. Our view: apply the same logic to people. The author of a change shouldn't be the only one to judge whether its tests are any good.
For tests, mutation testing gives a measurable check. Stryker's documentation describes the method: it injects bugs (mutants) into your code and reruns the tests. A mutant that makes a test fail is killed, and one that survives points at a gap. A suite that passes and lets mutants survive proves little. Mutation testing shows weak tests; it doesn't prove anyone understands the code.
4. Tier the risk
Not every change needs the same scrutiny. One commenter suggested marking agent-touched code as lower-trust and requiring human promotion after review. Our recommendation: authentication, money movement, deletion and permissions paths need a second reviewer who isn't the author and can explain the change, as our policy template also recommends. Everything else runs on rules 1 and 2.
| Rule | What it assigns | How it's enforced | Cost |
|---|---|---|---|
| One named owner | Who answers for a merge | CODEOWNERS plus required code owner review | Setup plus upkeep of the file |
| Explain-it gate | What the owner must be able to say | Pull request template, reviewer follow-up | A few minutes of author time per PR |
| Writer and checker apart | Who judges the tests | Second reviewer; mutation testing on critical modules | Review time, extra CI minutes |
| Risk tiers | Where a second explainer is needed | Branch rules on sensitive paths | Slower merges on the riskiest code |
5. Check it's working after a month
Pick five merged pull requests at random and ask the owner of each to walk you through it. If two can't, the gate is a form-filling exercise and you need to tighten it. Do the same after your next incident.
What to screen for in your next senior hire
Agent-loop skills are a separate screen, linked at the end of the next section. These probes test something different: whether the candidate understands code, with or without the tool. Our judgment, not validated predictors.
| Signal | Probe | Red flag |
|---|---|---|
| Judges AI output | "Walk me through a recent AI-assisted change. What did you reject, and why?" | Can't name a rejection |
| Reads unfamiliar code | "Explain this diff you didn't write." | Paraphrases the lines, can't say what breaks |
| Debugs unaided | Fix a failing test with the assistant switched off | Stalls without the tool |
| Uses AI to learn | "How do you use the tool: asking concepts, or handing over tasks?" | Hands over everything, can't describe what came back |
The debugging probe is motivated by the Anthropic trial, where the largest quiz gap was on debugging questions. The last row maps to its interaction patterns, with the caveats above. For the interview mechanics when candidates may use AI, see interviewing when candidates use AI.
What this means for your team shape and who owns merges
Four decisions, all judgment. First, name owners per code area before the next release; an area without one is where the next incident lands. Second, count review capacity honestly. One commenter on the first Hacker News thread described approving more diffs than they could read. Cap merge ownership per senior, not just pull request size, so nobody holds more than they can understand. Third, treat your next senior hire as a reviewer-owner seat and screen with the probes above rather than output speed. Fourth, keep juniors owning small merges with the explain gate. The Anthropic participants were mostly junior, and a gate gives them practice at the thing the study found lacking.
For what to interview for beyond comprehension, go to how senior engineers use AI coding agents.
FAQ
Who is responsible when AI-written code breaks production?
The author of record, and the reviewer who approved it. The tool isn't an owner. Osmani's argument is that code approved without full understanding "now carries implicit endorsement", so both names belong in the postmortem.
Should engineers be required to understand every line AI writes?
Every line is the wrong bar. Require the owner to explain what the change does, what breaks it and how to roll it back, plus the second-reviewer rule for authentication, money, deletion and permissions paths (rule 4 above). That's our recommendation.
What is comprehension debt?
Osmani's term for the distance between the code a system holds and what anyone understands. It grows when code is merged faster than people can read it, and healthy metrics can hide it.
How do you check an engineer understands code they merged?
Ask them to walk you through it without the prompt: what it does, what breaks it, how to roll it back. Do it on random merged pull requests, not only on ones that failed.
Does mutation testing prove code is understood?
No. Per Stryker's docs, it shows whether your tests catch injected changes. It tells you about test quality, which is useful, but a passing score says nothing about whether a person can explain the code.
Sources
- Anthropic Research: How AI assistance impacts the formation of coding skills
- Addy Osmani: Comprehension Debt, the hidden cost of AI generated code
- GitHub Docs: About code owners
- Stryker Mutator: What is mutation testing?
- Hacker News: Ask HN: How to deal with shipping AI-written code you don't understand?
- Hacker News: Ask HN: How to enforce engineers to understand the code they are shipping