AI agents: automating a process without breaking what already works
4 min read
Key takeaways
- RAG almost always fails at retrieval, never at generation.
- How documents are chunked is the decision with the greatest impact on answer quality.
- Combine semantic search and keyword search: neither is enough on its own.
- Always cite your sources: that is what makes the tool usable in a professional setting.
- Build your reference question set before writing the first line of code.
Plugging a language model into a folder of PDFs takes a day. The demo works, everyone is impressed. Three weeks later nobody is using it, because one answer in three is wrong and you never know which.
The problem is almost never the model. It lies in what you give it to read.
How it really works
A RAG assistant chains four steps on every question:
- Chunk the documents into passages, once and for all, upstream.
- Index these passages in a searchable form.
- Retrieve the passages relevant to the question asked.
- Generate the answer relying only on those passages.
The model only comes in at the last step. If retrieval brings back the wrong passages, no model in the world will rescue the answer. That is why almost all the effort must go into steps 1 to 3.
The thirty-second diagnosis: when an answer is wrong, look at the passages that were retrieved. In more than 80% of cases, the information requested was not there. The model did not lie: it answered with what it was given.
The six decisions that make the quality
1. How documents are chunked
This is the most important and most neglected decision. Cutting every 500 characters is simple and produces mediocre results: sentences are cut in the middle, context disappears, a table ends up broken into four unreadable pieces.
What works: cutting according to the structure of the document — by section, by article, by FAQ question — with an overlap of a few lines between neighbouring passages, and keeping the title of the original section as a header. A passage must remain understandable on its own.
2. Metadata
Each passage must carry its provenance: document, version, date, issuing department, confidentiality level. This is used to filter before searching — "only procedures currently in force", "only documents accessible to this department" — and to display the sources, which makes the tool credible.
3. Hybrid search
Semantic search finds different wordings but misses exact references: an article number, a product reference, an internal acronym. Keyword search does the opposite.
Use both and merge the results. On our projects, moving from purely semantic search to hybrid search improves relevance by 20 to 35%. It is the best effort-to-result ratio in the whole chain.
4. Reranking
Retrieve widely — twenty to thirty passages — then have these candidates re-ranked by a reranking model that judges the real relevance of each against the question. Keep only the best five for generation. Negligible cost, significant gain.
5. Generation instructions
Three sentences in the system prompt change everything:
- answer only from the passages provided;
- systematically cite the document and the section for each claim;
- if the passages do not contain the answer, say so explicitly rather than guessing.
6. The reference question set
Write fifty real questions, with their correct answers, before you start. They come from your users, not from your imagination. This set becomes your only objective way of knowing whether a change improves or degrades the system.
Without it, every change is an opinion.
+20–35%Relevance gained by moving to hybrid search
≈ 80%Of wrong answers caused by failed retrieval
50Reference questions to write before developing
The mistakes we see most
Ingesting the whole file share. Twenty years of archives, four versions of the same document, drafts that were never approved. The assistant then answers correctly… from a procedure repealed in 2019. Start with a restricted corpus that is up to date and has someone responsible for it.
Ignoring access rights. If the corpus contains documents that not everyone should see, rights-based filtering must happen before the search, never after. This is a point to deal with at design stage, not at the end of the project.
Confusing the demo with the product. A demo shows that the technology works. A product handles badly scanned documents, tables, ambiguous questions, answers that cannot be found, and scaling up. That is where five sixths of the work lies.
A document assistant is not judged on its good answers. It is judged on what it does when it does not know. A tool that says "I cannot find this information in the documents" is usable; a tool that makes things up is not.
A realistic schedule
| Period | Work | Result |
|---|---|---|
| Week 1 | Corpus selection, writing the question set | Document scope fixed |
| Weeks 2–3 | Chunking, indexing, hybrid search | First queryable version |
| Weeks 4–5 | Reranking, instructions, citations, measurement | Version evaluated on the reference set |
| Weeks 6–8 | Supervised use, corrections, access rights | Version deployed to a team |
Eight weeks, a controlled corpus, a measured quality indicator. That is what separates an assistant used every day from a forgotten demo.
If you are wondering which corpus to tackle first, our article on choosing a first AI use case gives the three filters we apply.
Sources
- Contextual retrieval in AI systems — Anthropic
- File search — OpenAI
- Citations — Anthropic
