RAG Explained: How Retrieval-Augmented Generation Works and How to Build It Well

Ask a general AI chatbot about your return policy or last quarter's campaign results and it will guess or say it does not know. The model was trained on public data up to a point in time and has never seen your documents.

Retrieval-augmented generation, or RAG, fixes that by looking up the most relevant passages of your own content and giving them to the model before it answers. The idea is simple. Building a RAG system that answers reliably is not, and the difference comes down to a handful of design decisions.

This guide explains how RAG works, then goes through the decisions that decide quality: how to chunk documents, how to search, how to rerank, how to evaluate, what it costs, a worked example and the failure modes to plan for.

What Is RAG?

RAG combines retrieval, which finds relevant information in a chosen set of documents, with generation, where a large language model writes an answer from that information. It was introduced in a 2020 research paper by Patrick Lewis and colleagues at Facebook AI Research and has become the standard way to connect AI to company knowledge.

How RAG Works

A RAG system has two pipelines. The indexing pipeline runs ahead of time; the query pipeline runs on every question.

Indexing pipeline

• Collect and clean documents: remove navigation, duplicates and outdated versions.
• Split them into chunks, small passages that can each answer part of a question.
• Turn each chunk into an embedding, a vector that captures its meaning.
• Store embeddings, chunk text and metadata such as source, date and access level in a vector database.

Query pipeline

• Embed the user's question.
• Retrieve the most similar chunks, filtered by metadata such as the user's permissions.
• Optionally rerank the results for relevance.
• Build a prompt with the question, the top chunks and instructions to answer only from them and cite sources.
• Generate the answer and show the sources.

Decision 1: How to Chunk Your Documents

Chunking has more effect on answer quality than most people expect. Chunks that are too large bury the relevant sentence in noise; chunks that are too small lose the context needed to answer.

• Split on structure first: headings, sections and list items, rather than a fixed character count.
• A common starting point is a few hundred tokens per chunk with a small overlap between neighbors, then adjust based on evaluation results.
• Keep each chunk self-contained: include the section heading or document title so the chunk makes sense on its own.
• Store useful metadata with each chunk: source URL, last updated date, product, region and audience.
• Handle tables and FAQs separately: one question and answer per chunk usually works better than splitting mid-table.

Decision 2: How to Search

• Vector search finds passages with similar meaning, even when the words differ.
• Keyword search, such as BM25, finds exact terms: product codes, names, error messages.
• Hybrid search combines both and is often more reliable than either alone, especially for business content full of specific names and codes.
• Metadata filters narrow the search before ranking, for example to the right product, language or permission level.

Decision 3: Rerank and Choose How Much Context to Send

A fast first search retrieves a wider set of candidates, then a reranking model scores each candidate against the question more carefully and keeps the best few. Sending fewer, better chunks usually beats sending many: it lowers cost, reduces the chance the model is distracted by marginal passages, and keeps you well inside the model's context window.

Decision 4: Write the Generation Prompt

• Tell the model to answer only from the provided passages.
• Tell it what to do when the passages do not contain the answer: say so and suggest a next step, rather than guess.
• Require citations to the passage or source for every factual statement.
• Set the tone, length and format your users expect.

How to Evaluate a RAG System

Evaluate retrieval and generation separately, so you know which part to fix:

• Build a test set of 50 to 100 real questions, each with the document that contains the answer.
• Retrieval hit rate: the share of questions where the right document appears in the top k results. If this is low, fix chunking, search or metadata before touching the prompt.
• Faithfulness: the share of answers where every claim is supported by the retrieved passages.
• Answer relevance: whether the answer actually addresses the question.
• Refusal accuracy: whether the system says “I do not know” when the answer is not in the documents.

Re-run the test set after every change to chunking, embedding model, search settings or prompt.

A Worked Example: A Customer Support Assistant

Imagine a store with 300 help center articles averaging 800 words. These numbers are illustrative:

• Chunked by section, the articles produce roughly 1,500 chunks, each tagged with article title, URL and product category.
• Search is hybrid, filtered by the product category the customer is asking about.
• The top 20 candidates are reranked and the best 4 are sent to the model, roughly 1,000 tokens of context per question.
• The prompt requires an answer from those passages only, with links to the source articles, and a handoff to a human when no passage answers the question.
• Before launch, 80 real past tickets are used as the test set. Retrieval hit rate in the top 4 shows whether the help center itself has gaps.

In projects like this, the first evaluation usually reveals content problems as much as technical ones: missing articles, outdated policies and two pages that contradict each other. Fixing the content often improves answers more than tuning the system.

What RAG Costs

• Indexing: embedding every chunk once, then again only when documents change.
• Storage: the vector database or managed search service.
• Per question: one embedding call, the search, an optional rerank call and the generation call. Generation input tokens grow with every chunk you send, so retrieving fewer, better chunks is the main cost lever.

RAG vs Fine-Tuning

Fine-tuning changes a model's behavior by training it on examples. RAG changes what information the model sees at answer time. Use RAG for facts that change or must be cited; use fine-tuning for consistent style, format or specialised behavior. Many production systems use both.

Failure Modes to Plan For

• Stale content: answers repeat an old policy because the old document is still indexed.
• Missed retrieval: the right passage exists but is not found, often because of poor chunking or missing keyword search.
• Conflicting sources: two documents disagree and the model picks one at random.
• Permission leaks: the system retrieves documents the user should not see. Apply access filters at retrieval, not after.
• Confident guessing: the model answers anyway when nothing relevant is retrieved. Instruct and test for refusals.

Left unchecked, these failures mean RAG reduces AI hallucinations less than teams expect, which is why evaluation matters more than any single tool choice.

RAG and AI Search

AI search engines work on the same principle: they retrieve web pages, then generate an answer from them. Content written as clear, self-contained passages that answer specific questions is easier to retrieve and cite, for your own RAG system and for AI search alike. Our guide to SEO, AEO and GEO covers how to structure content for that.

Where to Start

Pick one well defined document set and one audience. Many AI platforms now include built-in file search, so you can test whether RAG answers your real questions before building anything custom. Write the test set first, measure retrieval and faithfulness, fix the content gaps you find, then decide whether you need a custom pipeline.

If you want your team to understand and use tools like RAG with confidence, our AI training program can help. Contact us to plan a session.