We gave a talk at DevFest Istanbul this month on taking LLM prototypes to production — specifically, on the gap between a demo that impresses a room and a system that holds up on a Tuesday afternoon with real users and real data. That gap is longer than most teams budget for, and very little of it is model work.
What the talk covered
Four things, each drawn from projects we shipped over the last eighteen months, including the parts that went badly:
- Why an evaluation set is the first artifact worth building, ahead of any feature.
- How retrieval quality dominates model choice in most document-heavy applications.
- What monitoring looks like when the output is prose rather than a status code.
- The cost and latency work nobody budgets for and everybody eventually does.
The through-line was a single question: when this breaks in production, where will you look? A prototype never has to answer that. A system does, and the answer has to exist before the incident rather than be improvised during one.
The monitoring section drew the most questions, which surprised us. When the failure mode is a confident wrong answer rather than an exception, the usual dashboard reports that everything is fine. What we do instead is sample real traffic, score it against the same evaluation set the system was built against, and watch that number the way you would watch an error rate — while logging every retrieval, so a bad answer can be traced back to whether the right passage was in the context at all.
The questions afterwards
The room was mostly engineers who were already a prototype or two in, which made the discussion better than the slides. Several questions were organisational rather than technical: how to tell a stakeholder that the demo was the easy part, what to do when the evaluation set disagrees with the person who approved the project, how to price work whose hardest phase is invisible from the outside. We do not have clean answers to those, and said so.
The other recurring theme was when not to use a model at all. A surprising share of the problems described to us that evening were retrieval and workflow problems with a model bolted on, and would be cheaper, faster and more predictable as ordinary software. That is an unpopular thing to say at an AI talk, and it was the point several people wanted to keep talking about afterwards.
Thanks to the organisers, and to everyone who stayed to argue. The slides are not published, but if you were there, or the topic is close to something you are building, email us and we will send them over. We are also happy to walk a team through the same material at greater length.