A team ships an LLM demo in a week. Everyone claps. Then it takes nine months to get it into production — and half of those don't survive contact with real users. That gap between the demo and the pipeline is where most enterprise AI budgets go to die.
The prompt is the easy 10%. The other 90% is the plumbing, the guardrails, and the boring operational discipline that turns a clever autocomplete into something you can put your company's name behind.
The demo is a lie, and that's fine
A demo runs once, on a curated question, with a human ready to retry when it fumbles. Production runs ten thousand times a day, on inputs nobody anticipated, with no one watching. Those are different machines.
In my engagements, the teams that stall are the ones that treat "it worked in the playground" as evidence. It isn't. It's a hypothesis. The moment you accept that the model is a probabilistic component inside a deterministic system — not the system itself — the real work becomes obvious: everything around the model.
Retrieval is the product, the model is a commodity
Most enterprise use cases are not "be creative." They're "answer correctly from our data." That makes your retrieval layer the actual product, and the LLM a swappable engine underneath it.
Get the unglamorous parts right first:
- Chunking and metadata. Bad chunking quietly caps your accuracy no matter how good the model is. Tag chunks with source, date, and access level from day one.
- Freshness and permissions. If the answer cites a policy retired last quarter, you've built a liability. Retrieval must respect who is allowed to see what — the model has no concept of authorization.
- Evaluation on your data. Public benchmarks tell you almost nothing about your invoices, your contracts, your tickets. Build a golden set of 100–200 real questions with known answers, and score against it on every change.
Swapping the model is a config change. Fixing your data is a project. Budget accordingly.
Guardrails, cost, and the latency tax
A raw model call is not a feature. Between the user and the model, you need input validation, output checking, and a fallback for when the thing returns confident nonsense. Treat hallucination as a known failure mode you design around, not a bug you hope disappears.
Watch three numbers
- Cost per resolved task, not cost per token. A cheap model that needs three retries is more expensive than an expensive one that succeeds once.
- p95 latency. Averages hide the pain. Users feel the slow tail, and a two-second demo becomes an eleven-second production call once you add retrieval and safety checks.
- Escalation rate. How often does the system hand off to a human? That number is your real accuracy, and it belongs on a dashboard.
Caching, smaller models for easy paths, and routing hard cases to bigger ones will usually cut your bill more than any prompt trick. Optimize the pipeline, not the sentence.
Own the lifecycle, not just the launch
Models drift. Providers deprecate versions. Your users invent inputs you never imagined by Tuesday. An LLM feature is a living system, and it needs the same operational muscle as any other production service.
- Log everything — prompt, context, output, and user reaction — so you can debug the failure that happens at 2 a.m.
- Version your prompts like code. A prompt change is a deploy; it deserves review and a rollback path.
- Keep a human in the loop where the cost of being wrong is high. Automate the boring 80%; escalate the consequential 20%.
Where to start Monday
Pick one workflow with a narrow scope, a tolerant failure mode, and a measurable outcome — internal knowledge search, first-draft support replies, ticket triage. Ship it to a small group. Instrument it hard. Learn what actually breaks before you scale, because the failures you find in a pilot are cheap and the ones you find in production are not.
The organizations winning with LLMs aren't the ones with the cleverest prompts. They're the ones that built the pipeline around the prompt — and treated the model as the least interesting part of the problem.
If your AI initiative is stuck in the demo phase, the question isn't which model to use. It's which part of your pipeline you've been avoiding.