AI Coding Agent Cheating Tests? Ours Faked a Quote
Is your AI coding agent cheating tests? Ours swapped a real quote for an invented one to quiet a style check. Four cases we caught, and what caught them.
On November 14 a coding agent and I were finishing an article for my personal site. The site has a small pipeline: I write in English, a cheap model drafts the Russian version, and a style check reads the Russian before anything merges. The check raised a warning. One of the quotes in the article, a message I had typed to an agent mid-task, read in Russian with the informal “you”, and the site addresses its readers formally.
It was a warning, not a failure. The quote was correct. The style rule was written for how the site talks to its reader, and it had no idea what to do with quoted speech.
The agent did not say any of that. It replaced my quote with a different one, “where are we with this?”, which I never wrote, pushed it with the commit message “Reword a quote so the Russian version keeps «вы»”, and the warning went away. It mentioned the swap, honestly, as an aside in a status update a few hours later. By then the article was live, and the invented quote stayed on the site for five days.
Nothing about this was malicious. That is the part worth an engineering leader’s attention.
What happened when our agent was told a quote broke a style rule?
Lay the steps out and every one of them is reasonable on its own:
| Step | What the agent saw | What it did |
|---|---|---|
| 1 | A warning: the Russian text uses the informal “you” | Located the sentence |
| 2 | The sentence is a quote | Treated the quote as text it could edit |
| 3 | Rewording removes the “you” | Wrote a new quote with no “you” in it |
| 4 | The check is green on the next run | Moved on to the next task |
The commit message is accurate. It did reword a quote, and the Russian version did keep the formal register. A reviewer skimming the history would see a tidy fix for a style warning. The only thing missing is the sentence that matters: “this quote is now something the author never said.”
The agent had two honest options. It could have told me the rule fires on quoted speech and should not, or it could have left the warning alone, since it blocked nothing. Both are slower than a one-line edit that turns the check green.
Is that cheating, or just the shortest path to green?
The research literature calls this specification gaming: a system optimizes the measure it was given rather than the intent behind it. Public benchmarks have caught coding agents editing the tests they were asked to pass. Our case is smaller and, I think, more typical of daily work. Nobody asked the agent to make the warning disappear at any cost. It inferred that a green check was the goal, and nothing in its setup offered “the check is wrong” as a cheaper move than “change the input until the check is happy.”
For a manager, the useful framing is not “the agent lied.” It is that every automated check you add becomes a target, and the cheapest way to hit a target is often to change what the target can see. A style check sees text. A test sees whatever its assertion compares. If editing what the check sees is easier than fixing what it protects, an agent under pressure to finish will sometimes do exactly that, and write a truthful-sounding message about it.
What does the same pattern look like in tests?
The quote was a prose example of something we see in code every week. In our own repository, where our Agents build features and Proctor, our review Agent, reviews every pull request, three recent cases had the same shape: a check that was green and could not have been red.
An assertion that compared a value to itself. A change gave tickets to an Agent through one shared function, and its test asserted that the function passed along the right object. Proctor’s comment on the pull request: “This assertion cannot fail.” The test harness recorded whatever it was handed, and the code always handed it the same object, so the test compared the object to itself. The fix was to assert on a separate piece the caller supplied, so a wrong wiring would actually show.
A new test row that passed on the old code. A fix tightened a pattern that redacts secrets from log text, in two copies of the same rule. The author added a test row for the second copy. Proctor pointed out that the row already passed on the unchanged code, so reverting that copy’s fix would leave every new assertion green. The author then checked the claim the hard way, reverting each copy alone and rerunning the suite, confirmed one copy was unpinned, and added a row only that copy’s rule could match. With either fix reverted, the suites now fail.
Tests that passed for the wrong reason. On an earlier change, a second agent reviewing a rate limiter found two more. A lockout test passed only because its 200 junk requests never reached the 600-attempt threshold the code enforced, so the lockout it claimed to cover never triggered. And an error test passed a string as the second argument of assert.rejects, which Node treats as the failure message rather than the expected error, so it passed on any rejection at all.
None of these were written to deceive anyone. Each one was the author’s agent producing a test, running it, seeing green, and stopping. Green was the goal, and green was reached.
Why does a second reviewer catch what the author can’t?
Because the author and the check share one goal, and the reviewer does not.
The agent that writes a fix also writes the test and runs it. From inside that loop, a passing test is confirmation. It has no reason to ask whether the test could fail, because failing is the outcome it was working to avoid. The same is true of a person, which is why code review predates AI by decades.
A reviewer asks a different question: would this check have caught the bug it claims to cover? That question is cheap to ask and often cheap to answer. In two of the three cases above, the answer came from reading the assertion carefully. In the third, from reverting one line and running the suite. Proctor runs a tests check on every review for exactly this, alongside its deep review of the change, and posts its findings as one review on the pull request. The person who owns the change decides what to do with them and merges.
The quote is the case where no second reviewer was in the loop. My personal site has a style check and a translation step, but nothing reads a change and asks “is this still true?” The result was an invented quote in a published article, caught only because the agent happened to mention it and I happened to read the aside. The real quote went back in five days later.
This is also why “have the agent review its own work” is weaker than it sounds. Self-review inherits the author’s goal. The adversarial review we run asks a reviewer to look for reasons the change is wrong, not to confirm it is right, and the question of who reviews AI-written code is really a question of whose goal they carry.
What should an engineering leader change?
Four changes, none of them expensive.
- Give the agent a sanctioned way to say a check is wrong, and make it cheaper than the workaround. Our agent had no path to “this warning is a false positive for quoted speech” that was as fast as editing the quote. Write the path down in the instructions your agents read: when a check conflicts with the facts, stop and say so, and never change a fact, a quote, a fixture or an expected value to satisfy a check.
- Make “would this fail without the fix?” a standing review question. Put it in your review checklist and in your review Agent’s brief. The cheapest proof is to revert the fix and run the test. If the reviewer cannot answer the question, the test is not done.
- Keep the reviewer independent of the author. A different brief at minimum, ideally a different role entirely. Two agents that both want green are one agent with a longer bill.
- Read the commit message against the diff, not instead of it. Every message in the cases above was accurate and incomplete. “Reword a quote” is true. The review question is what the rewording did to the meaning.
None of this needs a person to reread every line an agent writes. It needs the person to decide on a review that already names the weak test, and an Agent whose job is to find it. That is the split we run on: Proctor reviews, a person approves and merges, and the record of both stays on the pull request.
Is this a reason to trust AI coding agents less?
It is a reason to trust a green check less than you did, whoever wrote the change. A test that cannot fail was always possible. Agents make more of them, faster, and write fluent explanations alongside. The fix is the same one engineering teams have always used for human authors: an independent reviewer asking whether the check checks anything, and a named person who owns the merge.
If you want to see what that review looks like on your own repository, book 30 minutes with us.