Retrieval-augmented generation, or RAG, supplies a language model with information retrieved from an external source when it answers a question. It is useful when an answer depends on business knowledge that is absent from, or newer than, the model’s training.
A working system needs more than a vector database. It needs reliable source documents, permissions, retrieval that finds the right passage and a way to check whether the answer is supported. Each of those can fail independently.
Follow the answer back to the source
Consider an employee asking which approval is needed for a new supplier. The application identifies the employee, searches the policies they can access and selects passages about supplier onboarding. Those passages accompany the question in the model request. The answer can then link to the policy and its relevant version.
This is an illustrative workflow, not a client result. It shows why source ownership matters. If the indexed policy is obsolete, a fluent answer with a real citation can still be wrong. If a table is split badly during extraction, the answer may apply a threshold to the wrong category.
The original RAG research combined a generative model with an external retrieval mechanism. Modern implementations vary: lexical search, structured database queries and hybrid retrieval can all supply evidence. A vector index is one option, rather than the definition of the architecture.
Evaluate retrieval before rewriting the prompt
Build a small, representative set of questions with the passages that should answer them. Include document titles people actually use, abbreviations, conflicting versions and questions that have no answer in the collection.
Inspect two separate outcomes:
- Did retrieval return relevant, authorised and current evidence?
- Did generation use that evidence accurately and avoid unsupported additions?
A prompt change cannot recover a passage that never reached the model. Increasing the number of retrieved chunks can add irrelevant material, duplicate text and cost. Reranking may help selection, but it introduces another component to evaluate.
Track source identifiers, document versions and retrieval configuration with each test. Otherwise a later quality change may be wrongly attributed to the language model when the index or document parser changed.
Permissions belong in the retrieval path
Apply access controls before restricted content reaches the model. Filtering links after generation is too late if the underlying text has already been included in the request.
Permissions also need to cover caches, shared conversation state and debugging tools. A response generated for one user must not become a reusable cache entry for another user who lacks access to its sources. Revoked access and deleted documents need an update path in the derived index.
Retrieved documents are data, including when they contain text that looks like an instruction. A malicious page or document can attempt prompt injection. Treat tool actions separately from the retrieved text and enforce authorisation in application code. OWASP’s prompt-injection guidance describes this class of risk and defence in depth.
Make uncertainty useful to the reader
A system should be able to say that the approved sources do not contain an answer. Define what happens when sources disagree, retrieval returns weak evidence or the question requires a professional decision.
Citations support inspection. They do not prove that the linked source supports every sentence or that the source itself is correct. For consequential work, test citation relevance and make the underlying passage easy to open.
Self-reported model confidence is not automatically a calibrated probability of correctness. Use observed performance on your own tasks, alongside explicit business rules, to decide where review is required.
Operate the knowledge system
Assign owners for source updates, access changes, index freshness and failed queries. Keep a regression set for the questions the system must continue to answer. Review retrieval and answer quality after changing an embedding model, parser, search configuration or language model.
RAG can make business knowledge available at the point of a question. The engineering work is making that connection inspectable and maintainable. See Knowledge Systems for how this fits a custom build.
Written by Raj Dalal, Advisor at Ryzolv.
Back to Insights