2 November 2026
·5 min read
Delivery & CI/CDacceptance criteria for AI agentsspecification by exampleA Spec Before Any Agent: Acceptance Criteria That Say Where Each Assertion Came From
Specification by example ties each acceptance criterion to a concrete example. For agent-assisted work, the spec should also record where each assertion came from, and no agent should start until that file exists.
Validation starts before the agent writes a line. When a coding agent turns a ticket into a pull request, the acceptance criteria decide whether the result is correct or merely plausible. If those criteria are vague, the agent fills the gaps with guesses, and the review becomes an argument about what the ticket meant. The fix is older than agents. Martin Fowler's Specification by Example ties every acceptance criterion to a concrete example, so the team agrees on behaviour in terms both a person and a machine can check. For agent-assisted work, one addition matters: the spec also records where each assertion came from.
What Specification by Example says
Fowler describes a working practice, not a tool. A team builds a shared understanding of what the software should do by writing concrete examples together. Those examples become the acceptance criteria. They are precise enough to execute, so they double as tests, and they stay readable, so they double as documentation. Product, testing and engineering agree on the examples first, and the examples drive what gets built.
Three things follow. Examples are concrete, not abstract rules. The same artefact serves as specification and as test. And the team agrees on behaviour before anyone writes an implementation.
Why agents raise the stakes
An agent is fast and literal. Give it a loose criterion like "the endpoint should handle invalid input gracefully" and it will produce something, run its own tests against its own reading of "graceful", and open a pull request that looks finished. The reviewer then has to reverse-engineer the intent, decide whether the agent's interpretation matches the real requirement, and do it without the shared conversation that would have happened if a person had written the code.
A concrete example closes that gap. "A POST with a negative quantity returns 422 and the body {"error": "quantity must be positive"}" leaves nothing for the agent to invent and nothing for the reviewer to guess. A machine can check the assertion before it reaches a human.
Examples alone do not close a second gap. When you read an acceptance criterion weeks later, you cannot tell whether an assertion encodes a real requirement or an assumption someone made while writing tests. For human-written code that ambiguity is survivable, because the author remembers. An agent remembers nothing, and neither does the next agent that touches the same code. So the spec needs provenance.
Record where each assertion came from
For each assertion in the acceptance criteria, write down its source. There are two kinds, and keeping them separate changes how you treat a failing check.
- A written requirement: the assertion traces to a product decision, a regulatory rule, a contract with another service, or an interface the team has committed to. If the code fails this check, the code is wrong.
- An observed behaviour: the assertion records how the current system happens to behave today, captured to avoid a regression. If the code fails this check, you have to decide whether the old behaviour was intended or incidental before you call it a bug.
The distinction matters most during change. An agent refactoring a module will trip assertions. If every assertion looks identical, the team either treats all of them as sacred, which blocks legitimate change, or waves them through, which lets regressions land. Tagging each assertion as requirement or observation tells the reviewer which failures are negotiable and which are not. It also tells the next agent that a requirement-backed assertion is a constraint to satisfy and an observation-backed one is a fact to question.
Three actions for this week
Pick one service your team is about to hand to an agent, and write the acceptance criteria as concrete examples before the agent starts. Use real request and response bodies, real status codes, real field names. If you cannot write the example, you do not yet know what you are asking for, and the agent will not either.
Add a provenance tag to every assertion in that spec. Mark each one as requirement or observation, with a one-line pointer to the ticket, the interface contract, or the test run that captured the current behaviour. Keep it in the same file as the criteria so it travels with the work.
Make the spec file a precondition in your pipeline. No agent run begins until the file exists and names a source for each assertion. This is cheap to enforce and it stops the most common failure, where the agent and the reviewer discover mid-review that they never agreed on what correct meant.
Where this sits in a delivery process
Specification by example is a team discipline, so it needs a place in the workflow rather than a document that rots. The reviewer checks the agent's output against the spec, reading the criteria and the provenance first and the diff second. A failing requirement-backed assertion is a blocker. A failing observation-backed assertion is a conversation. Both are visible before the code reaches a human, because the examples run as machine checks.
This also changes what handover means. A spec with sourced assertions tells whoever inherits the service which behaviours are promises and which are accidents of the current implementation. That is the difference between a test suite you trust and one you are afraid to touch.
At Anystack, every change we deliver starts as a written spec with acceptance criteria, passes machine checks before a human reads it, and carries a named reviewer of record. AI tools assist the engineers in the pod and never occupy a slot, so a person owns the spec and the review. You can see how the spec, the checks and the reviewer fit together in our delivery method.
*Anystack Engineering runs three-person engineering pods that deliver inside your repositories with a written spec for every change, a named reviewer of record and delivery measured per engagement. Engagements from £18,000 per month. How the pod delivers.*
