RAG Explained: How AI Apps Answer Questions From Your Own Documents

Ask a general chatbot about your company’s leave policy and it will either guess or apologise. It has never seen your HR handbook. Retrieval augmented generation, usually shortened to RAG, fixes that by letting the model look up relevant passages first and then write its answer from them.

It is the pattern behind most “chat with your documents” products, internal help desks and support bots.

The basic recipe

1. Collect the documents. PDFs, web pages, tickets, policies, product manuals.
2. Split them into chunks. A few hundred words each, often with some overlap so ideas are not cut in half.
3. Turn each chunk into an embedding. An embedding model converts text into a list of numbers that captures its meaning, so similar ideas land close together.
4. Store the embeddings in a vector database or a search index.
5. At question time, embed the user’s question, find the closest chunks, and paste them into the prompt.
6. Ask the language model to answer using only that material, ideally citing which chunk each claim came from.

Why not just train the model on the documents?

Fine tuning is expensive, slow to update and still does not guarantee accurate recall of specific facts. RAG keeps knowledge outside the model, so you can update a policy this morning and the bot uses it this afternoon. It also lets you show sources, which builds trust.

Where RAG goes wrong

• Bad chunks. If a table is split in the middle, the answer will be half right.
• Retrieval misses. The right passage exists but the search does not find it, often because users phrase things differently from the documents. Combining keyword search with vector search helps.
• Too much context. Stuffing twenty chunks into the prompt can bury the one that matters.
• Out of date sources. RAG faithfully repeats an old policy if the old file is still in the index.
• Hallucination anyway. Models can still fill gaps. Instruct them to say “I could not find that” when the sources are silent.

Tips if you are building one

• Clean your documents before indexing. Garbage in really does mean garbage out.
• Keep metadata such as document title, date and department, and filter on it.
• Test with real questions from real users, not just the ones you expect.
• Log questions that get poor answers and fix the content, not just the prompt.

The bottom line

RAG is less about clever AI and more about good information management. Get the documents, chunking and search right, and even a modest model will give grounded, useful answers.

Sources

• Lewis P. et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)