16 November 2026
·5 min read
QA & Testingcode coverage actionabilityvalidation in the age of AICode Coverage Actionability: What the Number Proves When a Model Wrote the Tests
A coverage percentage says which lines ran, not which failures your suite would catch. Google Research shows how to prioritise the next test by change risk and production context, which matters more when a model wrote some of the code.
Validating AI-assisted code runs into a measurement problem before it runs into a tooling one. When a model writes part of your code and part of your tests, the numbers on your dashboard carry less information than they used to. A coverage percentage is the first of those numbers to mislead you. It tells you which lines executed during the test run. It does not tell you which failures the suite would catch.
Google Research set out this gap in a paper presented at ICSE-SEIP 2024, Productive Coverage: Improving the Actionability of Code Coverage. The team prioritised uncovered code by its similarity to code that was already tested or frequently executed in production. In their environment this raised coverage and produced modest review-efficiency gains. The method matters more than the gain figure: decide where the next test goes by combining the coverage gap with change risk and production context, rather than chasing the percentage upward line by line.
What a coverage number proves
Coverage is an execution record. A line marked covered was reached by at least one test. Nothing in that record says the test asserted anything meaningful about the line, and nothing says a fault on that line would fail the build. You can reach 80% coverage with tests that execute code and check almost nothing. A model asked to raise coverage will write exactly those tests, because executing a line is cheaper than reasoning about what it should do.
That is the specific risk with generated tests. Example-based tests written by a model tend to encode the same assumptions as the code the model wrote. They run the lines. They agree with the implementation. The coverage number goes up. The suite's ability to detect a fault does not move, because the test and the code were drawn from the same understanding of the requirement.
What production context adds
Google Research offers a way to rank the uncovered code that is worth a test. Two signals do the ranking. The first is similarity to code paths that run often in production, a proxy for where a defect would be felt. The second is change risk, which points at code that has moved recently and is therefore more likely to carry a new fault. A line that is both frequently executed in production and recently changed is a better place for the next test than a cold, stable line that happens to be red on the coverage report.
This reframes the weekly coverage conversation. Instead of asking which files are below the threshold, the team asks which uncovered paths matter to users and which have changed under a model's hand. The percentage becomes an input to that decision rather than the target of it.
Three actions for this week
Start by separating the two numbers your dashboard conflates. Keep reporting line coverage, but add a measure of what the suite detects, not only what it executes. A mutation score is the direct version of that measure: it injects small faults and reports how many your tests catch. Run it on the modules that changed, so the cost stays bounded, and read it beside coverage rather than instead of it.
Second, rank the coverage gap by production context. Take your top executed paths from telemetry and your recently changed files from version control, and intersect them with the uncovered report. The lines in that intersection are where the next test earns its keep. This is a query you can write against data you already hold, not a new tool to buy.
Third, hold the tests for high-risk paths apart from the model that wrote the code. When the same agent can edit both the implementation and its tests, a green build proves only that the two agree. Have a person author the acceptance checks for the paths your production-context ranking flagged, derive them from the written requirement rather than the implementation, and keep them where the agent cannot modify them. The coverage number on those paths then means something, because a human decided what the test asserts.
Where this leaves the dashboard
The practical shift is small and it survives contact with a sceptical team. You are not throwing out coverage. You are refusing to let it stand alone as a validation signal when a model contributed to both the code and the tests. Coverage tells you reach. Mutation score tells you detection. Production context and change risk tell you priority. Read together, they point your scarce test-writing effort at the code where a fault would hurt.
For a team shipping agent-assisted changes at pace, the governing question is whether the next test you write would catch a real regression on a path your users touch. That question has an answer you can compute from data you already collect, and it changes which test gets written next.
How Anystack works with this
Anystack pods deliver inside your repositories and pipelines, and every change starts as a written spec and passes machine checks before a named reviewer of record sees it. For coverage, that means the acceptance checks on high-risk paths are authored and reviewed by a person and derived from the requirement, not generated from the diff. Our quality engineering work treats coverage as one input among several, read beside a detection measure and ranked by change risk and production context, so the number on your dashboard earns its place rather than flattering the suite.
*Anystack Engineering runs three-person engineering pods that deliver inside your repositories with a written spec for every change, a named reviewer of record and delivery measured per engagement. Engagements from £18,000 per month. How the pod delivers.*
