Property-Based Checks for Generated Code: Validation That Does Not Depend on the Example the Model Chose

The sharpest validation problem sits where a model writes the code and then writes the tests for it. The tests pass. The demo works. Nobody has checked the code against anything the model did not already believe.

Example-based tests encode specific inputs and the outputs the author expected. When the author is a model that wrote a faulty implementation, the examples it chooses tend to match the assumptions baked into that implementation. The test and the bug agree with each other. You get a green suite and a wrong answer.

Property-based testing attacks this from a different direction. Instead of listing example inputs, you state a property that must hold for a whole class of inputs, and the tool generates inputs from that property. Hypothesis, the Python library, does exactly this: you declare the shape of valid input and the rule the output must satisfy, and it searches for a counterexample, then shrinks any failure down to the smallest input that still breaks. The property comes from the requirement, not from the implementation the model produced. That independence is the whole point.

Why example-based tests inherit the model's blind spots

Consider a model asked to write a function that merges two sorted lists. It writes the merge, then writes three example tests: two short lists, one empty list, one list with a repeated value. All pass. The examples never include lists where the duplicate sits at the boundary between the two inputs, because the model's mental model of the function did not surface that case. If the merge mishandles that boundary, the suite stays green.

A property states the rule the examples skirted around. For a merge: the output is sorted, the output length equals the sum of the input lengths, and every element of the output came from one of the inputs. Hypothesis then generates thousands of input pairs, including the boundary duplicates, and reports the first pair that violates any of those three rules. The check did not depend on which examples the model thought to write.

Where properties are cheap, and where they are not

Properties are cheap to state when the function has a clear mathematical or structural invariant. Round-trip pairs are the easiest: encode then decode, serialise then parse, compress then decompress. The property is that the result equals the input. Sort, merge, dedupe and filter all have length and ordering invariants that take one line each. Parsers have the property that any string the parser accepts, the printer can reproduce.

Properties get expensive when the correct output depends on business rules that are themselves the thing in question. If the acceptance criterion is "apply the right discount", restating the discount logic as a property just rewrites the implementation in a second place, and a wrong model will get both wrong in the same way. For that kind of code, the independent check has to come from outside the model: a worked example from the requirement, a reconciliation against an authoritative source, or a reference implementation the model did not write.

Reach for a property when you can name an invariant that holds without re-deriving the business logic. When you cannot, write acceptance checks from the requirement by hand and keep them out of the model's reach.

What an engineering leader can do this week

Three concrete steps, each doable inside a sprint.

• Pick one module your team has let a coding agent touch in the last month, and find the functions with a clean invariant: round-trips, sort and merge, parse and print, set operations. Add a property-based test for each, written by a person from the requirement, not generated by the model.

• Write a one-paragraph standard that says any model-generated function with a stateable invariant gets a property test authored by someone who did not prompt the change. Put the standard where your reviewers see it during review.

• For the functions where a property would restate the business logic, require a worked example drawn from the written requirement and checked against an authoritative source. Record on the change where each assertion came from.

The second step matters most. If the same person prompts the agent and writes the property, the property drifts toward the implementation. Separating the author of the check from the author of the code keeps the validation independent. It is also why a named reviewer of record, someone who neither wrote nor prompted the diff, carries weight on generated changes.

How this fits into a delivery system

Property-based tests catch a specific failure mode: code and its example tests that agree because the same author wrote both. They do not replace acceptance checks from the requirement, mutation testing to measure what your suite actually detects, or human review of the diff. They sit alongside those as one more check the model's assumptions cannot quietly satisfy.

The pattern across all of these is the same. Put the validation on a path that does not derive from the implementation, and give it an owner who is not the author of the code. A property authored from the requirement, a reviewer who did not prompt the change, a reconciliation against authoritative data: each one breaks the loop where a model marks its own work.

Anystack's quality engineering work is built around that separation. Properties and acceptance checks are written from the requirement, machine checks run before human review, and a named reviewer who did not author the change signs it off. You can see how that validation path is structured in our quality engineering work, which lands inside your own repositories and pipelines rather than a separate QA silo.

*Anystack Engineering runs three-person engineering pods that deliver inside your repositories with a written spec for every change, a named reviewer of record and delivery measured per engagement. Engagements from £18,000 per month. How the pod delivers.*