AI-Written Code Needs an Independent Test Path: What the 14% Versus 25% Result Means for Your Suite

When a model writes both the code and the tests, the tests tend to agree with the code rather than check it. Konstantinou, Tambon and Papadakis report evidence for this in a July 2026 arXiv preprint (arxiv.org/abs/2607.05139, not yet peer reviewed). Across the studied tasks, tests generated after faulty model-written code detected those faults 14% of the time. Tests generated independently of the implementation detected them 25% of the time. That gap is the argument for keeping the test path separate from the code it is meant to catch.

The reason is mechanical. When an agent writes an implementation and then writes tests for it, the tests encode the same assumptions the implementation does. A test that asserts the code behaves as written will pass when the code is wrong in a way the model did not anticipate. The test inherits the bug. You get green checks that confirm the implementation matches itself, not the requirement.

What the study measured, and where it stops

The finding is narrow and worth stating precisely. The authors looked at fault detection rates for two kinds of generated tests on a set of tasks. Tests written after the faulty implementation detected the planted faults at 14%. Tests written independently detected them at 25%. The preprint has not been peer reviewed, and the number applies to the tasks studied, not to your codebase. It shows a direction: a test derived from an implementation is a weaker check of that implementation than a test derived from something else.

That "something else" is the practical lever. A validation path that does not descend from the code can come from the requirement, a contract, an interface definition, or observed behaviour of the system the code replaces. Where did the assertion come from, and did the code under test have any hand in writing it?

Three actions for this week

First, separate who or what writes the acceptance checks from who or what writes the implementation. If an agent produced the diff, the acceptance tests should not come from the same agent run against the same code. Derive them from the written requirement before the implementation exists, and make that ordering visible in the pull request. When the test file predates the implementation commit, a reviewer can see that the check was not reverse-engineered from the code.

Second, write assertions from the contract, not the function body. For an API change, assert against the published request and response schema. For a data transformation, assert against the documented input and output pairs. These assertions hold whatever the implementation does internally, so they fail when the implementation diverges from what was promised. The arXiv result suggests this is where the detection rate lives.

Third, keep the acceptance tests write-protected from the agent. If a coding agent can edit the tests that gate its own output, the retry loop will eventually loosen a failing assertion rather than fix the code. Put the acceptance suite in a path the agent cannot modify, and require a named human to change it. That keeps the 25%-style independent check from decaying into the 14%-style derived one over a few iterations.

A proof point from how review is supposed to work

Google's engineering practices guide describes a reviewer as responsible for the correctness and clarity of a change, not for rubber-stamping that it compiles. When a model generated the diff, the same logic applies to the test path. A reviewer checking AI-generated code should be able to answer where each test assertion came from and whether the code under test influenced it. If the agent wrote the test against its own output, the review has less to stand on, and the 14% figure is the reason.

This changes what a reviewer looks at. For agent-assisted changes, the review covers the provenance of the checks as much as the code itself. Did the acceptance criteria exist before the implementation? Are the assertions tied to the requirement or to the behaviour of the function the agent wrote? A principal engineer sampling a batch of these reviews can spot the pattern where tests and code share an author and treat those changes as under-validated.

What this looks like in a delivery pipeline

The habit worth building is cheap. Every change starts from a written spec that records what the code must do and where each acceptance criterion came from. The agent, or the engineer, then writes the implementation against that spec. Machine checks run the independent acceptance suite before any human looks at the diff. A named reviewer confirms the test path did not derive from the implementation. None of this requires new tooling; it requires ordering and a record.

Teams that skip the ordering tend to discover the problem the slow way. The suite stays green through a release, a fault reaches production, and the post-incident review finds a test that asserted the buggy behaviour as correct. The test was never wrong by its own logic. It checked that the code matched itself, and the code matched itself.

One caveat on scope. An independent test path raises fault detection; it does not reach 100% in the study, and it will not on your code either. Independent acceptance checks sit alongside the rest of your validation, not in place of it. Property-based checks, mutation testing to measure what your suite actually detects, and human review of the provenance each cover a different gap. The arXiv finding addresses one specific failure: tests that inherit the bug they were meant to catch.

Where Anystack fits

Anystack's quality engineering pods build the independent test path into how a change moves through your pipeline. Every change starts as a written spec that records where each acceptance criterion came from, those checks run before human review, and a named reviewer of record confirms the test path did not derive from the implementation an agent produced. The delivery meter stays with you at handover, so you can see which changes passed an independent check and which did not. AI tools assist the engineers in the pod and never occupy a seat, which keeps a human answerable for the provenance of every assertion.

*Anystack Engineering runs three-person engineering pods that deliver inside your repositories with a written spec for every change, a named reviewer of record and delivery measured per engagement. Engagements from £18,000 per month. How the pod delivers.*