AI Test Generation, With an Admissions Policy

Anyone can make AI write a thousand tests. The hard problem is deciding which of them deserve to exist.

The numbers on raw LLM test output are now well documented, and they are not flattering. TestPilot's authors found that only a median 61% of generated tests contain a non-trivial assertion — for one package, every generated test was trivial (Schäfer et al., IEEE TSE 2024). Roughly nine in ten passing generated tests add no coverage the suite did not already have. A 2026 study of LLM-generated tests for database systems found 96 flaky tests in generated suites where the human-written SQLite and DuckDB suites had zero. And in one systematic evaluation, barely a quarter of generated tests even compiled and passed against the code they were written for.

None of this means AI test generation is a bad idea. It means generation is the cheap half of the problem. Volume is free; meaning is not. What a serious engineering team needs is not a generator — it is an admissions policy.

Why coverage cannot be the gate

The reflex answer is to gate on coverage: admit the tests that raise it. The research killed that idea a decade ago. Coverage measures which lines ran, not whether any assertion would notice those lines going wrong — and coverage is not strongly correlated with a suite's ability to detect faults once suite size is controlled for (Inozemtseva & Holmes, ICSE 2014). The LLM era sharpened the point: researchers have produced suites with 100% line coverage and a 4% mutation score — a suite that touches everything and notices almost nothing.

The replacement currency is detection, measured directly: inject deliberate faults (mutants) into the code and observe which tests notice. Mutation score correlates with real-fault detection in a way coverage does not (Just et al., FSE 2014).

The admissions policy, gate by gate

Here is the standing rule our pods hold AI-written tests to. Each gate is a measurement, not an opinion, and they run in order — a flaky test corrupts every measurement downstream of it, so determinism goes first.

• Gate 1 — determinism. Run the candidate repeatedly (we default to 30 runs), in randomised order at unit level, and in parallel. Any variance rejects. The dominant flake class in generated tests is assertions over unordered collections — mechanically catchable, and worth catching before it poisons your kill matrix.

• Gate 2 — the gut check. Replace the implementation under test with an empty stub. The test must fail. A test that passes against a gutted implementation asserts nothing — it is mock choreography or a tautology. Run the complement too: the test must pass against the real code. Off-the-shelf extreme-mutation tooling does this today.

• Gate 3 — unique detection. Build the kill matrix: which tests kill which mutants. A candidate earns admission by killing something nothing else kills, counted in dominator mutants in place of raw kills. This is the anti-bloat gate stated as arithmetic.

• Gate 4 — redundancy. If the candidate's kill-set is a subset of what the suite already kills, it adds nothing. Reject. On raw LLM output this gate does most of the volume work.

• Gate 5 — boundary coupling. A test may depend only on the contract of the thing it targets, at its level. Unit tests may mock collaborators but may not assert internal call order; component tests touch the exported interface; system tests the public API. Tests coupled to implementation detail break on the next refactor and teach your team to ignore red.

• Gate 6 — cost. A runtime ceiling per admitted test. A meaningful test that takes ninety seconds is still a tax on every build.

Every admission and every rejection is recorded, with its reason, in a test admission ledger. The suite's meaningfulness stops being a claim and becomes a document you can read.

Two honesty clauses

The research forces two qualifications, and stating them plainly is part of the discipline.

First, the scope of the certificate. Mutation analysis presupposes the code currently behaves as intended — you inject faults into presumed-good code and ask whether the tests notice. So what these gates certify is regression protection. They do not certify the code is correct, and they cannot catch the ugliest failure mode in the literature: given buggy code, a model will sometimes write a test asserting the bug as intended behaviour. That test fails on a stub, kills mutants, and sails through every mechanical gate while entrenching the defect. Catching it requires asking where the assertion came from — the specification or the implementation — and that provenance check is human work.

Second, gate 3 is a prior, not a veto. In the foundational study, around 40% of tests that detected a real fault killed zero additional mutants. A hard unique-kill requirement would reject genuinely valuable tests. So a candidate that fails gate 3 but passes the gut check and anchors to a documented contract goes to quarantine for human review, not to automatic rejection. The same humility applies to gate 4: roughly a quarter of real faults couple to no mutant at all, so subsumption pruning is deliberately conservative around contract-anchored tests.

A rule is not a record

One more distinction we hold ourselves to. What you have just read is our standing rule — a commitment enforced on every engagement, backed by the published research cited above. It is not yet a claim about measured outcomes across engagements: that claim requires calibration against real-fault corpora such as Defects4J, and until those numbers exist we will not pretend otherwise. When we cite research, a study is named. When we state a rule, we hold ourselves to it. When we claim a record, a ledger exists behind it.

What to do this week

• Run extreme mutation (empty every method body) against one critical service and count how many of your existing tests still pass. Each one is a test asserting nothing — and if a machine wrote your recent tests, expect the number to be higher than you would like.

• Take the last batch of AI-generated tests merged into your codebase and rerun them thirty times, including in randomised order. Any variance is a flake you are about to start ignoring red builds over.

• If a vendor or an internal tool generates tests for you, ask one question: show me the rejection ledger. A generator with no admissions policy is a bloat machine, and a team that cannot show you what it rejected cannot show you what admission means.

The teams that get durable value from AI test generation over the next few years will not be the ones that generate the most tests. They will be the ones with the strictest door.

*Anystack Engineering ships qualified engineering pods — small, senior teams that deliver into your codebase with test-effectiveness measured by deliberate fault injection and every change adversarially reviewed before merge. Audits from £2,500. How verification delivers.*