Why AI Tools Fail in Real Use, Not Just in Testing
Most postmortems on a failed AI feature end up blaming the model — it hallucinated, it gave a bad answer, it didn't follow instructions. In our experience, the model is rarely where the actual failure originated. The failure is almost always in the system around the model: what it was given as input, what happened to its output, and what was supposed to catch the case where it went wrong.
A model that hallucinates a fact is doing exactly what a language model does. The systems failure is shipping that output directly to a user or a downstream system without a validation layer that checks it against something verifiable first. We treat every model output as untrusted input to the next step, the same way we'd treat any external API response — validate the shape, check it against known constraints, and have an explicit fallback for when it doesn't pass.
Retrieval is where we've seen the most production incidents that got blamed on the model but were actually retrieval bugs. A RAG pipeline that retrieves stale, irrelevant, or subtly wrong context will produce a confident, well-written, wrong answer — and a wrong answer that reads fluently is more dangerous than one that reads uncertain, because it's harder for a reviewer to catch. We spend more engineering time on retrieval quality than on prompt tuning, because retrieval bugs are silent in a way prompt bugs usually aren't.
Agentic workflows add a failure mode that a single model call doesn't have: compounding error across steps. An agent that's 95% reliable per step looks fine in a demo with three steps and looks completely unreliable in production once a real task takes fifteen steps to complete. We design agent workflows around explicit checkpoints where a human or a deterministic check can catch a wrong turn before it compounds through the next ten steps, rather than trusting the agent to self-correct indefinitely.
Cost and latency are systems problems that get discovered embarrassingly late. A workflow that calls a model three or four times per user action can look fine at ten test users and become unaffordable or unusably slow at real volume. We model the per-request cost and latency budget before building the workflow, not after it's already shipped and someone asks why the monthly bill tripled.
The teams that ship AI features successfully aren't the ones with access to the best model — model access is close to commoditized at this point. They're the ones who treat the model as one component in a system that also needs input validation, output verification, monitoring, and a clear answer to 'what happens when this is wrong,' the same rigor any other production system gets.
None of this is a reason to avoid shipping AI features. It's a reason to stop evaluating them the way you'd evaluate a chatbot demo, and start evaluating them the way you'd evaluate any other system that makes decisions with real consequences when it's wrong.
