9 November 2026
·5 min read
QA & Testingmutation testingvalidation in the age of AIMutation Testing in CI: What Your Suite Actually Detects
A coverage percentage says which lines ran. A mutation score says which faults your tests would catch. When a model wrote some of those tests, the difference decides whether your suite validates anything. Here is how to run mutation testing in CI without the cost getting out of hand.
When a model writes both the code and the tests, a passing suite proves the two agree, not that either is correct. You need a measure of what the suite would catch, not what it ran. Mutation testing gives you that measure.
A mutation tool such as Stryker injects small faults into your code, one at a time, then runs the suite against each altered version. If a test fails, the mutant is killed. If the suite stays green, the mutant survived, and you have found a change to your code that your tests do not notice. The mutation score is the share of injected faults the suite caught. Unlike a coverage percentage, it reports detection rather than execution.
Why coverage stops short, and the gap widens with generated tests
Code coverage tells you which lines ran during the suite. It says nothing about whether an assertion would have failed when the behaviour broke. A test can call a function, run every branch inside it, assert nothing meaningful, and still count as covered. That test adds a green tick and no protection.
This is a known weakness of coverage as a target. It becomes sharper when a model generates the tests. A model writing tests after the code tends to encode the same assumptions the code already makes, so the assertions describe what the code does rather than what the requirement wants. The lines are covered. The faults are not caught. Mutation testing separates those two states, because a surviving mutant is a concrete, reproducible change your assertions missed.
What a mutation run tells you that nothing else does
Three things come out of a run that are hard to get any other way.
- Which parts of the code your suite would not defend if an agent, a refactor or a bad merge altered them. Surviving mutants point straight at those lines.
- Whether a test file that looks thorough actually asserts anything. A method with high coverage and a low mutation score is asserting too little.
- A number that moves when the suite gets weaker, which a coverage percentage often does not. You can add shallow tests and watch coverage rise while mutation score stays flat.
The cost is the obvious objection. Mutation testing runs the suite once per mutant, so a large codebase produces a long run. That is why it does not belong on every push.
How to run it without the cost getting out of hand
Run mutation testing on a schedule, not per change. A weekly run against the main branch keeps the compute bounded and still catches a suite that has drifted. Most tools, including Stryker, support running only against files that changed since a given point, so you can scope a lighter run to recent diffs and keep the full sweep weekly.
Start with the code that would hurt most if it broke silently: payment logic, authorisation checks, data transforms that feed a system of record. Measure those first. A mutation score across your whole monorepo is a vanity number you will never act on; a score on the twenty files that matter is a work list.
Three steps an engineering leader can take this week:
- Pick one high-risk module and run a mutation tool against its tests once, by hand, off the critical path. Read the surviving mutants. Each one is a test you are missing or an assertion that is too loose.
- Add a weekly scheduled mutation run in CI scoped to that module and two or three others, with the report published where the team reviewing changes can see it. Do not gate merges on it yet; gating a noisy new signal breaks trust in it.
- Set a rule that any test file a model generated gets a mutation check before it is trusted as a guard. A generated test with a low mutation score is documentation of current behaviour, not validation.
What a mutation score does not tell you
Two honest limits. First, a mutant that survives can be equivalent, meaning the injected change does not alter observable behaviour and no test could ever catch it. These are false alarms you triage out, and they take reviewer time. Second, a high mutation score on a weak set of mutant operators still leaves gaps the operators never probed. The score measures detection against the faults the tool injects, nothing beyond that. Treat it as one meter among several, read alongside production incident history and the risk of the change in front of you, not as a single pass-fail gate.
The pattern underneath all of this is to keep validation independent of whatever wrote the code. Mutation testing does that by generating the faults itself, so the check does not inherit the assumptions of the implementation or the tests. When some of your tests came from a model, that independence is what you are buying.
How Anystack helps
Our quality engineering work puts measurement like this where a team can act on it: a scheduled mutation run on the modules that carry real risk, surviving mutants triaged to a named owner, and a delivery meter you keep at handover so the signal does not leave with us. Every change we make starts as a written spec and passes machine checks before a human reviews it, so the tests that guard your code are held to the same standard as the code itself. You can see how that fits a suite you already run on our quality engineering page.
*Anystack Engineering runs three-person engineering pods that deliver inside your repositories with a written spec for every change, a named reviewer of record and delivery measured per engagement. Engagements from £18,000 per month. How the pod delivers.*
