Mutation Testing Is the Meter of What Your Suite Detects

Validation with AI has a quiet failure mode: a model writes the code, the same model writes the tests, coverage reads high, and nobody can say what the suite would catch. Line coverage records which statements ran while the tests executed. It does not record whether a test would fail if the code were wrong. The gap between those two widens once generated tests encode the same assumptions as the generated code they check.

Mutation testing measures detection directly. A tool such as Stryker injects small faults into your source, one at a time, then runs your suite against each mutant. A mutant that makes a test fail is "killed". A mutant that survives is a change your tests did not notice. The mutation score is the share of injected faults your suite detected.

What a mutation score tells you that coverage does not

Consider a function that applies a discount and a test that asserts the result is a positive number. Run that test and the function's lines are covered. Now change the discount operator from multiply to divide. The result is still positive, so the test still passes. Coverage was 100 percent; the detection was zero for that fault. Mutation testing would report the surviving mutant and name the line.

Three things follow for an engineering leader deciding where to spend review time.

First, a high coverage number on a model-generated suite is weak evidence. The tests ran. Whether they assert anything a bug would violate is unknown until something perturbs the code. Mutation testing is the perturbation.

Second, the surviving mutants point at specific lines, not at a percentage. A survivor on a pricing branch is a different risk from a survivor in a log-formatting helper. The report lets you read the map and decide which gaps matter.

Third, the score is comparable over time within one codebase. You can watch whether detection on a given module holds as the module changes, which is the signal you want when agents are committing diffs into it.

The cost objection, and the answer

Mutation testing is expensive by construction. For N mutants it runs the suite N times, so a naive run on a large repository can take hours. That cost is why teams try it once, flinch, and shelve it. The fix is scheduling, not abandonment.

Run mutation testing weekly rather than on every change. A per-change gate forces the full cost into the critical path of every pull request and trains people to route around it. A weekly run on the main branch keeps the cost bounded and still catches detection gaps while they are days old rather than quarters old. Stryker supports incremental runs that focus on changed files, which narrows the weekly job further once you have a baseline.

The weekly cadence also changes what the number is for. A per-change threshold becomes a thing to satisfy, and satisfying it tends to produce assertions that kill mutants without checking behaviour anyone cares about. A weekly report read by a named engineer becomes a thing to act on: which modules lost detection this week, and why.

What to do this week

Pick one module where correctness carries real cost. Billing, entitlement, tax, anything where a silent wrong answer reaches a customer or a ledger. You want a module small enough to finish a run and important enough that the result changes a decision.

Run Stryker against that module once, outside CI, and read the surviving mutants rather than the headline score. Each survivor is a concrete sentence: this fault would ship undetected. Decide per survivor whether it needs an assertion or is genuinely not worth catching. Write that decision down; the percentage on its own is not the point.

Schedule the run weekly on the main branch, scoped to the modules you care about, with the report delivered to a person by name. Set a floor you will not cross for those modules rather than a target you chase everywhere. A floor on the paths that matter beats a repository-wide average that hides exactly the gaps you need to see.

If some of the tests in that module were generated by a model, treat the mutation report as the first independent check on them. Generated tests that share the code's assumptions tend to leave operator and boundary mutants alive, because the code and the test agree on the wrong thing. Those survivors are where to spend a reviewer's attention.

Where this sits in a validation path

Mutation testing is one instrument, not a programme. It measures what a suite detects; it does not tell you whether the suite tests the right requirement. Pair it with a validation path that does not derive from the implementation, such as acceptance checks written from the requirement before any code or test is generated. The mutation score then tells you how well those checks, plus the rest of the suite, resist actual faults.

For teams shipping agent-assisted diffs, the sequence is: a written spec defines the behaviour, acceptance checks assert it independently of the code, machine checks run before a human looks, and a weekly mutation run meters whether the whole suite still catches injected faults. Each step answers a question the previous one cannot. Coverage answers none of them on its own.

The weekly number will move. When it drops on a module, the report names the lines, and the named engineer decides whether the drop is acceptable. That decision, recorded against the change, is what a reliability-minded buyer is paying for when they ask whether the tests are any good.

Anystack's quality engineering work builds this kind of meter into a client's own pipeline: mutation runs scoped to the paths that matter, scheduled to keep cost bounded, with a named owner reading the survivors rather than the headline. The same engagement sets up the independent validation path the score depends on, so a rising coverage figure on generated code never stands in for detection you have not measured. You can see how we structure that in quality engineering.

*Anystack Engineering runs three-person engineering pods that deliver inside your repositories with a written spec for every change, a named reviewer of record and delivery measured per engagement. Engagements from £18,000 per month. How the pod delivers.*