Why 45% of Our AI-Generated Tests Were Paper Shields: Auditing Our Own Code with Pramāṇa

The green build illusion

Engineering teams adopting AI coding assistants frequently encounter a deceptive milestone: pull requests arrive accompanied by extensive test suites, line coverage hits 95%+, and CI pipelines run entirely green.

In practice, a green test suite generated by an LLM frequently acts as a paper shield.

Because language models generate tests by observing the implementation code, they naturally mirror implementation assumptions. They write tests that check whatever the code executes. They introduce loose matchers that swallow missing fields, test happy paths exclusively, and ratify existing bugs under the guise of verification. The tests do not validate requirements; they ratify the code as written.

We decided to measure this phenomenon directly on our own codebase.

The subject: Auditing Anystack's automation policy

We pointed Pramāṇa — our two-stage AI test validation engine — at our internal policy module: - Source Module: src/automation/policy.ts (governing outbound message validation, human approval gates, idempotency hashing, CRM retry exponential backoff, and procurement eligibility). - Existing Test Suite: src/automation/__tests__/policy.test.ts (14 AI-generated Vitest tests, passing 100% in CI).

To standard CI and standard code coverage tools, this module appeared thoroughly protected.

Then we executed the two evaluation stages of Pramāṇa.

Stage 1: The Static Assertion Substance Scan

Stage 1 runs instantaneous static AST analysis across the source and test files. It identifies domain error branches and status variants defined in the source, cross-checks whether any test expectation asserts their exact values, and flags loose matchers that mask corrupted data.

The scan returned immediate warnings: - Domain error / status branches in source: 30 - Asserted in test suite: 14 (47%) - Completely unasserted in test suite: 16 (53%) - Loose partial matchers: 4 (expect.arrayContaining)

The AI-generated tests had left major control branches unasserted: 1. invalid_word_count: The test never checked drafts under 45 words or over 120 words. 2. banned_language: The test never tested whether prohibited buzzwords or sycophantic greetings triggered errors. 3. not_approved: Missing approval timestamps were never tested in isolation. 4. deadline_passed & deadline_invalid: The procurement gate never verified handling of expired deadlines. 5. calculateCrmRetry: The exponential backoff calculations and dead-letter queue transition (attempts >= 5) had zero test assertions.

Stage 1 revealed that over half the domain-critical error conditions were never verified by the existing suite.

Stage 2: Targeted Mutation Proving

Stage 1 inspects assertions statically. Stage 2 proves behavior dynamically through systematic fault injection.

Pramāṇa generated 67 candidate semantic mutations across policy.ts — inverting logical operators (|| to &&), modifying relational boundaries (< to <=), and altering status branches.

If a test suite genuinely defends system integrity, injecting a defect must cause at least one test to fail ("killing" the mutant). If every test stays green despite the defect, the mutant survives — proving that the test suite is blind to that failure mode.

The baseline results: - Total candidate mutants evaluated: 67 - Mutants killed: 37 (55%) - Mutants survived: 30 (45%) - Verdict: REJECT AI TEST SUITE (45% False Confidence)

Nearly half the introduced defects went completely undetected: - Inverting the word count bounds check from || to && passed all tests. - Removing the CRM dead-letter limit passed all tests. - Loosening fee currency validation passed all tests.

The surviving mutant locations mapped directly to the unasserted branches flagged in Stage 1.

Hardening: Driving Fault Resistance to 100%

Using Pramāṇa's surviving-mutant line traces, we systematically hardened policy.test.ts: 1. Replaced loose expect.arrayContaining helpers with strict array equality checks (expect(result.errors).toEqual([...])). 2. Added boundary test cases for minimum (45) and maximum (120) word counts, confirming that 44 and 121 fail. 3. Added exact zero-age boundary tests for observation and verification timestamps (observedAt: base.now). 4. Added exhaustive test coverage for exponential backoff steps (5, 10, 20, 40 minutes), the 5th-attempt dead-letter transition, and RangeError validation.

When we re-ran Pramāṇa: - Domain branches asserted: 30 of 30 (100%) - Unasserted branches: 0 - Loose matchers: 0 - Total mutants evaluated: 67 - Mutants killed: 67 (100%) - Mutants survived: 0 (0%) - Verdict: ACCEPT (100% Fault Resistance)

The core lesson for engineering leaders

When software teams adopt AI agents, the primary constraint shifts from authoring code to verifying evidence.

Conventional line and branch coverage metrics only confirm that an execution path was traversed. They provide zero proof that an assertion would fail if the business logic produced an erroneous outcome.

Pramāṇa operates on an uncompromising principle: The model proposes, the machine proves. We never rely on an LLM to evaluate its own code or judge its own tests.

By running Stage 1 static assertion scans on pre-commit and Stage 2 targeted mutation proving in pull-request validation, engineering organisations can eliminate false test confidence and ensure that AI-authored code is backed by verifiable proof.