Why LLM-Generated Tests Fail on Your Worst Code (And Why That Matters)

The pitch for LLM-based test generation is that it works everywhere: point an agent at a file, get tests back. A study published this month, Code Health in LLM-Based Test Generation: Effectiveness and Token Efficiency, complicates that story. The authors measured how the effectiveness of LLM-generated unit tests varies with the maintainability of the code under test — using CodeScene's Code Health metric — across Python, Java, and C++, and scored the results with both coverage and mutation testing.

The finding is one every engineering leader running an AI-assisted testing pilot should internalise: LLM test generation is not maintainability-neutral. The tests get weaker, and more expensive to produce, precisely on the code that is hardest to maintain — which is usually the code carrying the most risk.

What the research actually found

Three results are worth translating out of the paper's language.

• Effectiveness tracks Code Health. As the maintainability of the source code drops, the mutation score of the generated tests drops with it. Coverage alone hides this — a test suite can execute a tangled function line-by-line and still fail to catch a mutation that changes its behaviour. Mutation score exposes the gap.

• Token cost rises as code health falls. Low-health code — long functions, deep nesting, high complexity — produces larger, messier inputs, which means more tokens consumed to generate tests that are, per the first finding, less effective. You pay more for less.

• The effect holds across languages. Python, Java, and C++ all show the pattern. This is not a quirk of one ecosystem or one model's training distribution; it is a property of how these tools reason about code.

None of this says LLM test generation is useless. It says the value is unevenly distributed, and the distribution is the opposite of what you want: strongest on clean, well-factored code that was already easy to test, weakest on the gnarly legacy modules where a good test suite would actually change your risk profile.

Why this matters for the buyer, both the researcher

Most enterprise codebases are not uniformly healthy. There is a long tail of files that everyone is afraid to touch — the billing calculator, the permissions resolver, the state machine that three people understand. Those files are low Code Health almost by definition, and they are exactly where an untested change becomes an incident.

If you roll out AI test generation as a blanket policy and measure success by coverage percentage, you will get a number that goes up while your real exposure barely moves. The tool will happily generate confident-looking tests for the dangerous modules, they will pass, and they will catch nothing — because a test that pins in the current (possibly buggy) behaviour of a tangled function still shows green.

This is the same trap as measuring test suites by line coverage generally, but AI makes it worse: it removes the friction that used to signal "this code is hard to test, maybe fix the code first." The agent doesn't get frustrated. It just produces output.

Three things to do this week

• Score your AI-generated tests with mutation testing, not coverage. Pick one service, run your generated suite through a mutation tool (Stryker for JS/TS, PIT for Java, mutmut for Python), and compare mutation score to line coverage. The gap between them is your false-confidence budget. If coverage is 85% and mutation score is 40%, most of your "tested" code is not actually protected against regressions.

• Segment your codebase by maintainability before you segment by coverage. Run a Code Health or complexity scan and cross-reference it with change frequency and incident history. The files that are low-health, high-churn, and incident-prone are where AI test generation will underperform most — and where human-written characterisation tests earn their keep. Route AI at the healthy code and senior attention at the rest.

• Treat low Code Health as a precondition to fix, not a target to test around. The research's uncomfortable implication is that the most effective way to get better AI-generated tests for a bad module is to refactor the module first. Break the 300-line function into named, testable units, and both the generated tests and their mutation scores improve. Refactor-then-generate beats generate-on-mess.

A proof point from adjacent research

This lines up with a broader move toward evidence-based admission of machine-generated artefacts. Benchmarks like OdinEval for LLM program repair insist that a candidate fix must make a failing regression test pass before it is admitted — the machine's output is not trusted on its say-so, it is verified against an executable bar. The same discipline applies to generated tests: a test is only worth keeping if it demonstrably fails when the behaviour it claims to protect is broken. Mutation testing is how you check that at scale. The consistent lesson across this literature is that AI accelerates production of plausible artefacts, and the burden shifts to verification.

Where this fits in a delivery process

The operational answer is not "stop using AI for tests" and it is not "trust the coverage number." It is to build a verification gate that measures whether tests are effective against your bar, and to route generation and human effort by maintainability. That is exactly the mechanism a qualified engineering pod applies inside delivery: test-effectiveness measurement — mutation scoring, both coverage — and adversarial review, so what reaches production is evidenced against the client's standard in place of counted as green ticks.

Concretely, what 90 days with a senior pod looks like on this problem: an initial Code Health and mutation-score baseline across the services in scope; AI-assisted generation aimed at the well-factored code where it is genuinely effective, with every generated test validated by mutation testing before it enters the suite; and senior engineers writing characterisation tests for the low-health, high-risk modules before refactoring them, so coverage and confidence move together. The AI is a tool the pod uses in its own delivery to go faster on the easy 70% — it does not replace the judgement about where the real risk sits.

The headline for engineering leaders is simple. AI test generation is a genuine accelerator, but it is strongest where you need it least and weakest where you need it most. Measure effectiveness with mutation testing, segment your codebase by maintainability, and fix the worst code before you ask a model to test it. Anystack's approach to quality engineering and test automation is built around that verification-first stance — evidence that tests catch real defects, not a coverage number that flatters the dashboard.