Explain retrieval-augmented generation
Model answer
Retrieval-augmented generation (RAG) gives a language model relevant documents at query time, so its answer is grounded in your data instead of only its training memory. It reduces hallucination, lets you use private or fresh knowledge without retraining, and makes it possible to cite sources.
Indexing (offline). Split documents into chunks, turn each chunk into an embedding vector and store it in a vector index, together with metadata.
Query (online). Embed the user question, retrieve the top-k most similar chunks (often combined with keyword search and a reranker), and put them into the prompt with an instruction to answer only from the context. The model then generates the answer.
Quality depends mostly on retrieval: chunk size, embedding model, hybrid search, metadata filters and reranking. Evaluate retrieval (is the right chunk in the top-k?) separately from generation (is the answer faithful to the context?).
RAG does not remove hallucination completely, and it adds latency and an index to maintain.
Key points
- Retrieve relevant chunks at query time and put them in the prompt
- Indexing: chunk, embed, store in a vector index with metadata
- Hybrid search and reranking improve retrieval quality
- Evaluate retrieval and generation separately
Likely follow-ups
- How do you choose a chunk size and overlap?
- When would fine-tuning be better than RAG?
Common mistakes
- Believing RAG eliminates hallucination
- Tuning only the prompt while retrieval returns the wrong chunks