TL;DR
RAG (Retrieval-Augmented Generation), proposed by Lewis et al. in 2020, retrieves relevant passages from an external knowledge base before an LLM generates an answer. It addresses hallucination, stale knowledge cutoffs, and private data. The standard pipeline: chunk documents → embed into vectors → store in a vector DB → retrieve Top-K at query time → optionally rerank → stuff into context → generate.
Pipeline
| Step | Detail | Common tools |
|---|---|---|
| 1. Chunking | Split docs into semantic chunks | LangChain, LlamaIndex |
| 2. Embedding | Convert chunks to vectors | OpenAI, BGE, text-embedding models |
| 3. Storage | Vector database | pgvector, Milvus, Qdrant, FAISS |
| 4. Retrieval | Similarity search Top-K | Vector + keyword hybrid |
| 5. Rerank | Precision boost (optional) | Cohere Rerank, BGE-Reranker |
| 6. Generation | Stuff passages into the prompt | Any LLM |
RAG vs plain LLM vs fine-tuning
| Approach | Update cost | Hallucination risk | Private data | Best for |
|---|---|---|---|---|
| Plain LLM | None | High (cutoff) | No | General Q&A |
| RAG | Low (swap docs) | Medium (retrieval-dependent) | Yes | Support, enterprise KB, live data |
| Fine-tuning | High (training) | Medium | Learnable but fragile | Tone, format, domain terms |
Common pitfalls
- Chunks too large: retrieved passages carry noise and waste context; too small: semantics break;
- Vector-only search fails on IDs/product codes; use hybrid keyword+vector retrieval;
- Skipping rerank: top-K often contains noise;
- Hallucination is reduced, not eliminated: wrong/stale retrieved docs get confidently quoted. Always cite sources and evaluate.
FAQ
RAG or fine-tuning?
Choose RAG for fast updates, document-grounded answers, and private data; choose fine-tuning to change tone, output format, or domain vocabulary. Production often combines both.
What chunk size should I use?
There is no single answer; 300–800 tokens is a common range. Aim for "one chunk can answer one question" and tune with retrieval experiments.
Does RAG eliminate hallucination?
No. It reduces hallucination substantially, but retrieval can still return wrong documents; use source citations and evaluate hit rate and faithfulness.
Sources
- Lewis et al., RAG for Knowledge-Intensive NLP Tasks, arXiv:2005.11401 (2020), accessed 2026-08-04
- IBM / AWS official RAG documentation, accessed 2026-08-04