We spent a good part of this year building a retrieval-augmented system that answers questions over a client's internal documents. It is live, it is used daily, and almost none of what made it work was the part people ask about first.
The headline lesson: retrieval quality beats model choice, and it is not close. We swapped the generation model three times during the project and the answers barely moved. We changed how documents were split and indexed once, and the failure rate dropped by more than half. If the right passage is not in the context, no model recovers it. If it is, a mid-sized model usually does fine — at a fraction of the cost of the one you were about to reach for.
Build the evaluation set before the features
We started with roughly eighty real questions, collected from the people who would use the system, each with an answer a domain expert had agreed was correct. It felt like a slow week. It was the highest-leverage week of the project.
Without an evaluation set, every change is an argument about impressions. Someone tries five queries, likes what they see, and ships. The next person tries five different queries and wants it reverted. With a set, you can say that a chunking change moved recall from 71% to 84%, and the debate ends in a sentence.
It also tells you when to stop. Teams keep tuning long after the remaining errors stopped being retrieval problems at all — the passage was there, the model had it, and the answer was still wrong because the question was ambiguous or the source document contradicted itself. No amount of index tuning fixes those, and the eval set is what makes the distinction visible.
Boring chunking wins
We tried semantic chunking, sliding windows with generous overlap, and a hierarchy of summaries. What shipped was fixed-size chunks that respect document structure, with the section heading prepended to each one. It is unglamorous, and it beat every clever alternative we tested on our own set.
Our read is that the sophisticated methods solve a problem most document sets do not have. When sources are well structured — headings, sections, a table of contents — the structure already carries the semantics, and preserving it is cheaper and far more predictable than inferring it. Reach for the clever approach after you have measured that the simple one is what is failing.
The last stretch is plumbing
The end of the project was not modelling at all. It was permissions, so nobody retrieves a passage from a document they are not allowed to open. It was stale content, because a system that confidently cites last year's policy is worse than one that says nothing. It was three file formats that parsed badly, and a sync job that had to survive its source system going down for an afternoon without quietly serving a half-built index.
None of that demos well. All of it is the difference between a prototype and something a company will put in front of its own staff, and it is reliably where the schedule goes.