Shadow Mode Before Go-Live: Validating an AI Workflow Against the Process It Replaces

A workflow can read well in a demo and then write the wrong value into a system of record on its third day live. One control catches that before it happens. When an AI workflow is about to update a system your business depends on, run it in shadow first, compare its decisions against the process it replaces, and let it act only when the record supports it.

The pattern is not new. Google's SRE book chapter on testing for reliability describes canarying and staged rollout: you expose a change to a slice of real traffic, watch it against the known-good baseline, and expand only when the signal holds. For an AI workflow that will write to a system of record, the equivalent is a shadow run that observes and compares but does not act.

Why a demo is not evidence

A demo shows the workflow handling inputs someone chose. Production sends inputs nobody chose. The gap between those two is where the cost lives, and it stays invisible until real volume flows through.

The SRE argument for staged rollout is that a change can pass every test you thought to write and still fail on the traffic you did not anticipate. An AI workflow inherits that risk and adds its own. Its decisions depend on document layouts, message phrasings and edge cases that shift over time, so a single evaluation run captures one moment, not the range the workflow will see.

Shadow mode replaces the demo with a running comparison. The workflow processes the same live inputs as the current process, produces its decision, and records it. Nothing reaches the system of record. You then read the disagreements.

What the shadow run measures

Three signals come out of a shadow run, and each one maps to an action you can take this week.

• Agreement rate: how often the workflow's decision matches the current process on the same input. Low agreement early is expected; the question is whether it climbs as you tune.
• Disagreement content: how often the two differ, and where. A workflow that disagrees on a narrow, identifiable class of inputs is easier to trust than one that disagrees at random.
• Failure to decide: inputs the workflow could not process or returned with low confidence. These tell you the size of the exception path before you commit to it.

For a leader reviewing a shadow run, read the disagreements by hand, not just the headline agreement number. A 95% agreement rate hides two different situations: one where the 5% is a coherent group you can write a rule for, and one where it is scattered and unpredictable. The first is ready to move forward with a guarded exception path. The second is not.

Set the promotion bar before you start

Staged rollout only works if you decide in advance what would let the change proceed. Otherwise the shadow run becomes a source of comfortable numbers that never quite triggers a decision.

Write the bar down before the shadow run begins. It should name the agreement threshold, the classes of disagreement that must be resolved rather than tolerated, and the person who owns the sign-off. This mirrors the SRE practice of defining what a canary must show before it expands, rather than eyeballing dashboards and hoping.

A workable bar for an AI workflow that writes to a system of record has three parts. First, agreement above a stated threshold on the full shadow window, not a cherry-picked slice. Second, every disagreement class either explained by a written rule or routed to a named exception owner. Third, a low-confidence path that a person actually staffs, because the workflow will hand off cases and those cases cannot sit unattended.

The action record is the other half

Shadow mode tells you whether to go live. Once you do, you need to know what the workflow did and be able to reconstruct any single decision. That is a separate control from the shadow run, and it matters more once the workflow is acting on its own.

An independent action record captures each decision the workflow made, the input it acted on, the confidence it reported and the rule or path that governed the outcome. Independent means the record is not written by the same component that made the decision, so a fault in the workflow does not silently corrupt its own audit trail. When a value in the system of record looks wrong three weeks later, the record traces it back to the input and the decision rather than leaving you to guess.

Here the low-confidence path and the exception owner reconnect. A workflow that routes conflicting or unclear items to a named person, with the source attached, keeps a human in the decision for the cases that warrant one. The record shows which items were routed, who resolved them and how, so the exception path is auditable rather than a black hole.

A sequence you can start this week

If you have an AI workflow that is due to touch a system of record, the steps are concrete.

• Write the promotion bar first: agreement threshold, disagreement classes that must be resolved, named sign-off owner.
• Run the workflow in shadow beside the current process on live inputs, recording its decisions without letting it act.
• Read the disagreements by class, not just the aggregate rate, and decide whether each class is rule-shaped or random.
• Stand up the exception path and the independent action record before go-live, and name the person who owns exceptions.
• Promote to acting only when the shadow record clears the bar you wrote.

Each step produces evidence you can show a sceptical stakeholder. That is why you borrow the staged-rollout discipline: the decision to go live rests on a record, not on a demo and a good feeling.

How Anystack helps teams act on this

Anystack builds controlled AI automation for operational workflows across documents, messages, forms, spreadsheets, APIs and system events, inside your own Google Cloud project. Each workflow ships with validation gates, a named exception owner and an independent action record, so a shadow run has somewhere to write its comparison and a live workflow leaves a trail you can audit. To see how the gates and the action record fit together before any workflow touches a system of record, the mechanism is described on our automation page.

*Anystack Engineering runs three-person engineering pods that deliver inside your repositories with a written spec for every change, a named reviewer of record and delivery measured per engagement. Engagements from £18,000 per month. How the pod delivers.*