Why Your LLM Feature Passed the Demo and Failed in Production

The demo worked. The stakeholder nodded. Three weeks later the feature is in production and the support queue is filling with hallucinated answers, the cloud bill has jumped without explanation, and nobody can say whether last Tuesday's prompt change made things better or worse. This is the defining engineering pain of 2026, and it is remarkably consistent across teams.

The gap is almost never the model. FutureAGI's field write-up on building LLMs for production (futureagi.com/blog/building-llms-production-2025) makes the point bluntly: the same model that impressed in a controlled demo fails in production because the surrounding engineering — evaluation, cost control, and graceful degradation — was never built. A demo is a single happy-path trace. Production is thousands of adversarial, malformed, and edge-case inputs hitting a non-deterministic system with no test to catch regressions.

Below are the three failures that account for most stalled LLM projects, and what to do about each this week.

Failure 1: No evaluation harness, so you cannot tell if a change helped

Traditional software has a pass/fail test. LLM features often have neither. Teams ship a prompt, eyeball a few outputs, and move on. When they later change the prompt, swap the model, or adjust retrieval, they have no way to know whether quality went up or down — because there is no scored, versioned set of inputs and expected behaviours to run against.

The result is change paralysis. Nobody wants to touch the prompt because nobody can prove the new version is better, and every deploy is a coin flip. This is the same class of problem that flaky, unmeasured test suites create in conventional code — except worse, because the output is fuzzy and human review does not scale.

What to do this week: build a golden set. Collect 50–100 real inputs from your logs, including the ones that failed. Define, for each, what a correct or acceptable response looks like — even if the check is a rubric scored by a second model or a simple assertion (contains the right entity, cites the right source, refuses when it should). Run this set on every prompt or model change and record the score. You do not need a fancy platform to start; a script and a spreadsheet beat vibes. This is the LLM analogue of measuring test effectiveness rather than counting green ticks, which is the discipline our quality engineering practice applies to conventional code.

Failure 2: No cost instrumentation, so the bill surprises you

LLM cost is a function of token volume, model choice, retries, and context size — all of which move in production in ways they never do in a demo. A retrieval step that stuffs 8,000 tokens of context into every call, a retry loop that fires three times on failure, or a switch to a larger model for one edge case can multiply cost silently. Because the spend is buried in an API bill rather than a per-request metric, nobody notices until finance asks.

The teams who avoid this treat token cost like latency: a per-request budget, measured, with alerts. The same operational rigour that keeps platform reliability and cloud cost under control applies here — you cannot manage what you do not instrument.

What to do this week: log token counts (input and output) and estimated cost per request, tagged by feature and user segment. Set a cost-per-request budget and alert when a deploy breaches it. The first time you look at this data you will almost certainly find a context-stuffing step or a retry storm inflating spend by a multiple. Fixing those two is usually the highest-ROI work available.

Failure 3: No fallback logic, so one failure becomes a user-facing outage

LLM APIs time out. They rate-limit. They occasionally return malformed JSON that breaks your parser. In a demo none of this happens. In production it happens daily, and if your feature has no fallback path, a transient upstream failure becomes a broken experience for the user.

A production-grade LLM feature degrades gracefully: it has a timeout budget, a retry policy with a ceiling, a schema validator on the output, and a defined behaviour when the model is unavailable — a cached answer, a simpler deterministic path, or an honest 'we couldn't answer that right now' rather than a spinning loader or a stack trace. Cloudflare's own post-incident discipline of failing small — containing a component failure so it does not cascade into a full outage — is the same principle applied at platform scale: assume the dependency will fail and design the blast radius down.

What to do this week: map every failure mode of your LLM call — timeout, rate limit, malformed output, empty retrieval — and define the user-facing behaviour for each. Add a schema validator on structured outputs and a hard timeout. Decide what happens when the model is down and build that path explicitly rather than letting the exception propagate.

The underlying pattern: production readiness is engineering, not prompt-tuning

What these three failures share is that none of them is about the model or the prompt. They are about the harness around it — exactly the surface area teams skip when a demo creates false confidence. A feature that passes a demo has proven it can produce one good answer once. Production readiness means proving it produces acceptable answers under load, within budget, and degrades safely when things break.

This is where an honest ownership question matters. When a senior engineer reviews a stalled LLM project, the most revealing question is: who owns the evaluation layer? If the answer is 'the person who wrote the prompt, informally', the feature is not production-ready regardless of how good the demo looked. Someone has to own the golden set, the cost budget, and the failure catalogue as living artefacts, not one-off setup.

A useful sequencing, in order of impact:

• Build the golden set and score every change against it — this ends change paralysis and makes improvement measurable
• Instrument token cost per request and set a budget with alerts — this catches the silent spend before finance does
• Catalogue failure modes and build explicit fallback paths — this turns transient upstream failures into non-events instead of outages

None of this requires a research team. It requires the discipline to treat an LLM feature as a production system with non-deterministic behaviour, and to build the evidence that it works before, not after, it reaches users.

How Anystack approaches this

When a client's LLM feature has stalled between demo and production, a senior engineering pod installs the harness first: a scored evaluation set drawn from real traffic, cost instrumentation per request, and explicit fallback logic for every failure mode — then works on the prompt and retrieval. The pod uses AI in its own delivery — retrieval over the client's docs to build the golden set faster, drafting failure-mode catalogues from existing incident logs — but the deliverable is evidenced production readiness against the client's bar, not a better demo. The measurement mechanism is the same one applied to conventional code: what reaches production is checked against a defined standard, adversarially reviewed, and owned by someone after the pod leaves.