29 October 2026

·

6 min read

AI in QAAI coding agent retriesvalidation gates

The Third Failed Attempt: Retry Budgets for AI Coding Agents and When a Person Takes Over

An AI coding agent that retries against a failing gate will keep going until something stops it. A written retry budget, with a named engineer who takes the task over at the limit, keeps the loop bounded and the attempt count on the record.

Anystack Engineering

Validation with AI covers whether a check passes and what happens when a check keeps failing and the thing producing the code is a loop that does not get tired. Anthropic's guide to building effective agents describes evaluator-optimiser patterns, where one agent generates a response and another evaluates it against feedback, then the first tries again. The pattern works when the evaluation is clear and iteration improves the result. It has no natural stopping point. Left alone against a gate it cannot satisfy, an agent will try, fail, adjust, try again, and keep going until something external tells it to stop.

For an engineering leader, the risk is not the first failure. It is the fifth, the tenth, and the drift that accumulates across them. Each retry can move the code further from the original intent as the agent chases the gate rather than the requirement. The fix is a written limit on how many attempts an agent gets, a named engineer who takes the task over when that limit is reached, and an attempt count that lands in the record for the change.

What the evaluator-optimiser loop actually does

Anthropic describes the loop as two roles. One generates, the other evaluates and returns feedback, and the cycle repeats until the evaluation is satisfied. The guide notes this works best when you have clear evaluation criteria and when iterative refinement measurably helps. Those are real conditions, and they both assume the loop ends.

In practice, a coding agent iterating against a test suite or a linter is running this pattern. The gate is the evaluator. The agent is the generator. When the gate is a well-formed assertion the code can be made to satisfy, the loop converges. When the gate encodes something the agent cannot work out from the context it has, the loop does not converge. It produces a sequence of plausible-looking changes, each one green on some checks and red on the one that matters, until a budget or a human intervenes.

The method depends on criteria being clear and refinement helping. When either condition breaks, the loop is no longer making progress, and no amount of further iteration will fix that. Someone outside the loop has to notice.

Three things to put in place this week

*Set a retry budget per task and write it down.* Decide how many attempts an agent gets against a given gate before the task stops being an agent task. Three is a defensible default for most coding work, because the first failure is information, the second is a correction, and a third failure against the same gate usually means the problem is not one more iteration away. The number matters less than the fact that it is written and enforced rather than left to whoever is watching the terminal. Put it in the same place you keep your CI configuration, so it is version-controlled and visible.

*Name the engineer who takes over at the limit.* A budget with no owner is a budget that gets quietly raised. When the agent exhausts its attempts, the task moves to a named person, not back into the queue for another run. That person reads the attempt history, decides whether the gate is wrong or the approach is wrong, and either fixes the task or fixes the gate. Here a human judgement call replaces mechanical iteration, and it needs a name against it so it happens.

*Record the attempt count on the change.* Every change the agent produced should carry how many tries it took to get there. A change that passed on the first attempt and a change that passed on the third tell you different things about the gate, the task description, and where your spec was thin. Over a few weeks the attempt counts show where your requirements are unclear enough that an agent cannot satisfy them without help.

Why the attempt count belongs in the record

A passing build hides its own history. The green tick looks the same whether the code arrived cleanly or after ten rounds of an agent grinding against a gate. For code a person wrote, the review captures some of that history in the discussion. For code an agent wrote across multiple retries, the history disappears unless you keep it on purpose.

Keeping the attempt count turns the retry budget into a signal. A task that regularly hits its limit tells you the gate is ambiguous, the task description is underspecified, or the work is genuinely hard and should have gone to a person first. None of that is visible from the final diff. All of it is visible from the attempt history, and all of it points at where your validation is weaker than the passing build suggests.

The same principle applies to any AI-assisted change: the output needs an independent record of how it got there, with more than confirmation that it arrived. A retry count is the cheapest version of that record for coding agents, and it costs you one field.

Where this fits with the rest of your validation

A retry budget does not replace the gates. It bounds the loop that runs against them. You still need assertions worth passing, a reviewer who reads the diff rather than the tick, and tests the agent did not write for itself. The budget sits underneath all of that and stops one specific failure mode: an agent that converts a bad task or an ambiguous gate into an unbounded sequence of changes nobody asked for.

The handover point is where this earns its place. When the budget is exhausted and a named engineer picks the task up, that person does the work the loop could not: deciding whether the problem is the code, the gate, or the requirement behind both. That decision needs a human, and it needs to be logged as one. An agent reporting its tenth failed attempt is not asking for an eleventh. It is telling you the task left its lane several attempts ago.

The discipline is cheap to adopt and hard to retrofit after an agent has already drifted. Write the number, name the owner, keep the count. Three fields, enforced before the agent starts, and the loop can no longer run away from you.

How Anystack approaches this

Our pods run agent-assisted work inside your repositories with the loop bounded from the start. Every change begins as a written spec, passes machine checks before human review, and carries a named reviewer of record. When an agent exhausts its attempts against a gate, a named engineer takes the task over, and the attempt history lands on the delivery meter you keep at handover. The AI tools assist the engineers; they never occupy an engineer slot. You can see how the gates and the reviewer of record fit together in our quality engineering work.


*Anystack Engineering runs three-person engineering pods that deliver inside your repositories with a written spec for every change, a named reviewer of record and delivery measured per engagement. Engagements from £18,000 per month. How the pod delivers.*

Discussions

0

Technical analysis, failure mode challenges, and reproduction observations.

Add to discussion

0/3000

No discussions yet. Share an observation or technical question above.

Start a conversation

Share the engineering context and delivery objective when you are ready to discuss the work.

Contact Anystack →

See the evidence

Read selected engineering work and its provenance.

Browse selected work →