What is a RAG chatbot? Retrieval-augmented generation in plain words

A RAG chatbot uses retrieval-augmented generation: when someone asks a question, it searches your website and documents for the passages that match, adds them to the AI model's prompt, and has the model write its answer from them. The model does not memorise your site. It reads the relevant pieces each time.

The term comes from a 2020 paper by Patrick Lewis and colleagues, accepted at NeurIPS 2020. This guide covers what the paper set out to fix, how the idea works in a website chatbot, using the model makers' and cloud providers' own documentation read on 13 October 2026, and where it goes wrong.

Where retrieval-augmented generation comes from

The paper starts from a limit of large language models: they “store factual knowledge in their parameters”, but “providing provenance for their decisions and updating their world knowledge remain open research problems.” Its answer was to give a model an outside memory, in its case “a dense vector index of Wikipedia, accessed with a pre-trained neural retriever.” The authors found that RAG models “generate more specific, diverse and factual language” than a model working from its parameters alone.

Those are a business's two problems with a chatbot too. The model knows nothing about your opening hours, and when a price changes you want the answer to change with it, with a way to see where it came from. OpenAI's accuracy guide sums RAG up as “the process of Retrieving content to Augment your LLM's prompt before Generating an answer.”

How a RAG chatbot answers from your website

Anthropic's engineering post on contextual retrieval describes the usual setup. In a website chatbot it runs in six steps:

1. Collect. The chatbot reads your pages, sitemap and files, and any questions and answers you write.
2. Chunk. It breaks the text into smaller pieces, “usually no more than a few hundred tokens”, so a search can return the paragraph that matters instead of a whole page.
3. Index. An embedding model turns each chunk into a vector, a list of numbers that encodes its meaning, stored in a database that can be searched by similarity. Some systems also keep a keyword index.
4. Search. When a visitor asks something, the question is matched against the index to find the most relevant chunks.
5. Prompt. The top few chunks go into the prompt, with instructions on how to use them.
6. Answer and cite. The model writes the reply, and the chatbot can show which documents it drew on.

A follow-up such as “and how much is that one?” means little on its own, so some systems rewrite it into a full question before searching. OpenAI's retrieval guide has a setting for exactly that, rewrite_query.

Keyword search, vector search, or both?

Vector search finds passages by meaning. OpenAI's retrieval guide says semantic search surfaces similar results “even when they match few or no keywords”, so a question can find a passage worded quite differently. Its weak spot is exact terms. Anthropic's example is a support search for “Error code TS-999”: an embedding model “might find content about error codes in general, but could miss the exact ‘TS-999’ match”, which a keyword method such as BM25 finds.

Hybrid search runs both and merges the two lists, often with Reciprocal Rank Fusion. Microsoft's Azure AI Search documentation explains it: each result scores 1/(rank + k) in each list, the scores are added, and documents near the top of both lists rise. It says the method performs best with a small k, “such as 60”. A reranking step can then score the merged candidates against the question once more; Azure runs its semantic ranker after the fusion.

Why chunks lose their meaning

Cutting a page into pieces can strip out what a piece is about. Anthropic's example is a chunk that reads “The company's revenue grew by 3% over the previous quarter”, with nothing to say which company or which quarter. On a business site the same happens to a line like “includes 100 GB storage” once it is separated from the heading that named the plan. Anthropic's fix is to add a short note of context to each chunk before indexing it, and it reports that this cut failed retrievals by 49%, or by 67% when combined with reranking. You can help from your side: write headings that name the subject, and keep each fact next to the thing it describes.

Citations: showing where an answer came from

Provenance was one of the problems RAG set out to solve. Google's grounding overview lists auditability among the benefits, through grounding support, “which are links to sources.” Citations can work two ways. The model can quote and cite the passages it used, which ties each claim to a passage. Or the system can list the documents it retrieved for that answer, which is simpler and shows where the material came from, though not which sentence came from which page. Either way, a visitor can open the page and check.

Where RAG goes wrong

OpenAI's accuracy guide splits failures in two: retrieval can supply “the wrong context, so the model can't possibly answer”, or “too much irrelevant context, which drowns out the real information”, and the model “can also get the right context and do the wrong thing with it.” For a website chatbot that usually means one of these:

• Stale pages. The index holds last month's copy of your site. Ask how often pages are re-read, and re-read a page by hand after an important change.
• Missing content. If no page says it, nothing can be retrieved. A good chatbot then says it does not know, and its list of unanswered questions shows you what to write.
• The wrong chunk. A passage that answers a nearby question gets used for this one: the Basic plan's limits given for Pro, or the UK returns policy for an EU customer.
• The right chunk, used badly. The model mixes two passages or adds a detail of its own. Instructions to quote figures exactly, and sources a visitor can check, are the guard. Our post on why AI chatbots make things up covers the rest.

Do you need RAG at all?

Not always. Anthropic's post says that if a knowledge base is smaller than 200,000 tokens, “about 500 pages of material”, you can put all of it in the prompt with no retrieval step. Retrieval becomes the practical choice once your content is larger than that, changes often, or needs answers tied to a source.

How MagizAI's chatbot uses RAG

MagizAI's chatbot reads your pages, sitemap, files, questions and answers and plain text. What happens next depends on whether the chatbot has a key for an embedding model, OpenAI by default or Google Gemini. With one, it splits each source into chunks of about 1,000 characters, at headings first and never through a table, attaches the page title and heading path to each chunk, and at question time runs a vector search and a whole-word keyword search side by side, merged with Reciprocal Rank Fusion at k = 60. Without one, it searches the stored text of each source by whole words and takes an excerpt of about 1,200 characters around the match.

Either way, a follow-up is rewritten into a full question first, and a question in another language can also be searched in your content's language. When there are more candidates than fit, a fast model scores each from 0 to 10 against the question and drops anything under 4. Up to five passages go into the prompt, up to three sources are shown under the answer, and web pages are re-read weekly by default. The training guide shows how to add content, connecting an AI provider covers keys, and the website chatbot page covers the rest.