What is RAG? Retrieval-Augmented Generation Explained

Every language model has a knowledge cutoff — a date beyond which it has no information. Ask GPT-5 about something that happened last week and it genuinely does not know. Ask it about your company's internal documentation and it has never seen it. RAG — Retrieval-Augmented Generation — is the standard solution to both problems.

How RAG works

• The user asks a question
• A retrieval system searches a document database for relevant chunks — typically using semantic (vector) search
• The top matching chunks are inserted into the model's context window alongside the question
• The model generates an answer grounded in the retrieved documents, rather than relying on training data alone
• The response can cite specific sources, enabling verification

The retrieval step is powered by embeddings — numerical representations of text that capture semantic meaning. Similar sentences end up close together in embedding space, so searching for 'employee leave policy' will surface relevant HR documents even if they use different phrasing. Most production RAG systems use a vector database (Pinecone, Weaviate, pgvector) to store and search these embeddings efficiently.

When to use RAG — and when not to

RAG is the right choice when you need a model to answer questions about specific, frequently-updated, or private information. Customer support bots, internal knowledge bases, and document Q&A tools are classic RAG use cases. Where RAG struggles is with tasks requiring deep reasoning across many documents simultaneously — the retrieval step can miss relevant context, and too many chunks in the context window dilutes the model's focus. For those tasks, fine-tuning or a large-context model may serve better.

Why it matters for how AI is actually used

RAG is the reason AI assistants can be useful inside enterprise environments without retraining models on sensitive data. Instead of teaching a model your proprietary information permanently (expensive, slow, and a potential security issue), you keep your data in a searchable store and retrieve it on demand. This architecture is now the foundation of most serious AI applications built on top of foundation models.