The Gap Between Demo and Production
Building a ChatGPT wrapper that impresses in a demo takes an afternoon. Shipping an LLM-powered feature that works reliably for 10,000 users a day — handling edge cases, controlling costs, meeting latency budgets, and not hallucinating on your most sensitive use cases — is a fundamentally different problem.
Over the last 12 months, our AI team has deployed LLM-powered features into six production environments: a legal contract analyser, a healthcare triage assistant, an e-commerce product description generator, a customer support deflection bot, a financial report summariser, and a code review assistant. Here is what we learned.
1. Prompt Engineering is Engineering
The most common mistake is treating prompts as an afterthought. On one healthcare engagement, our first-iteration prompt for clinical note summarisation achieved 73% accuracy on our test set. By the time we shipped, after four weeks of iterative prompt engineering, structured output specifications, and few-shot examples, accuracy was 94.1%. That 21-point improvement came entirely from prompt work — no model change required.
Our prompt engineering process: (1) Define evaluation criteria before writing a single prompt. (2) Build a dataset of representative inputs with human-labelled expected outputs. (3) Iterate systematically — one variable at a time. (4) Version control every prompt in Git. (5) Run automated regression tests on every prompt change.
2. RAG is Not Optional for Domain-Specific Use Cases
Base LLMs hallucinate facts. For any application where factual accuracy matters — legal, healthcare, finance — Retrieval-Augmented Generation (RAG) is not optional. Our legal contract analyser uses a hybrid retrieval pipeline: dense retrieval via a fine-tuned sentence transformer (legal corpus), sparse retrieval via BM25 for keyword precision, and a re-ranking step using a cross-encoder. The retrieval stack alone reduced hallucination rate from 8.2% to 0.4% on our eval set.
3. Latency Management: Stream Everything
GPT-4 Turbo averages 2–4 seconds to first token on complex prompts. For any user-facing feature, perceived latency matters enormously. Implement streaming responses from day one — users tolerate waiting for tokens to appear character-by-character far better than staring at a spinner. Internally, we use Server-Sent Events (SSE) to stream from the OpenAI API through our Laravel backend to the React frontend. This dropped perceived latency by ~65% in user research sessions.
4. Cost Control: Cache Aggressively, Batch Where Possible
LLM API costs compound at scale. On one e-commerce project, naive implementation was generating 40,000 product descriptions per day at GPT-4 pricing — a $3,200/day inference bill. After implementing semantic caching (cache hits for semantically similar inputs using vector similarity), batching similar requests, and switching to GPT-4o-mini for 70% of calls, we brought the daily cost to $340 — a 89% reduction with no measurable quality degradation on that task.
5. Observability: You Cannot Improve What You Cannot Measure
Every LLM call in production should log: model, prompt hash, input tokens, output tokens, latency, cost, and output. Build an evaluation pipeline that periodically samples production outputs and scores them against your rubric. We use a lightweight LLM-as-judge approach for automated eval, with human review flagged when judge confidence is low. Langsmith, Langfuse, and Helicone are all solid options for LLM observability — choose based on your architecture.
Key Takeaways
LLMs are powerful but require engineering discipline: treat prompts as code, build evaluation frameworks early, implement RAG for domain-specific accuracy, stream responses for UX, and instrument everything. The teams that win with LLMs are not the ones with the flashiest demos — they are the ones with the most systematic engineering practices around model evaluation and production operations.
