What is RAG (Retrieval-Augmented Generation)?

Retrieval-Augmented Generation (RAG) is the production pattern most B2B teams should understand before funding another AI pilot. It connects a frozen large language model to your private knowledge at query time: documents are chunked, embedded, indexed, and retrieved when a user asks a question.

The model does not memorize your policies in its weights. It reads relevant paragraphs supplied in the prompt and generates an answer grounded in that context. For EU organizations, that separation matters: you can refresh the index when GDPR notices change, filter by data subject in metadata, and show citations in the UI without retraining.

Why RAG exists and what problem it solves

General-purpose models know public internet statistics — not your internal policies, SKU tables, or client contracts. Fine-tuning can encode behavior but rots when facts change weekly. RAG keeps facts in a search index you control and keeps the model as a reasoning layer on top.

This article is part of our AI cluster. For the strategic comparison with fine-tuning, read the pillar guide on RAG vs fine-tuning.

• RAG vs Fine-Tuning — pillar guide
• Cost comparison

The four-layer architecture in production

Production RAG is four cooperating systems, not a single API call. Ingestion parses PDFs, HTML wikis, tickets, and CRM exports into clean text with metadata such as product line, locale, effective date, and access group.

Indexing embeds chunks with a chosen model and stores vectors plus metadata — often alongside keyword indexes for hybrid search. Retrieval runs on every query: embed the question, fetch candidates, optionally rerank, apply filters. Generation sends the user message, system instructions, and retrieved chunks to the LLM with rules to refuse when context is insufficient.

Layer responsibilities

• Ingestion — scheduled sync, OCR, PII redaction before indexing.
• Storage — Pinecone, Qdrant, pgvector, or Weaviate with tenant namespaces.
• Retrieval — hybrid BM25 plus vectors; rerank top candidates.
• Generation — citations, structured output, human escalation paths.

Embeddings and chunking — where quality is won

Teams underestimate chunking. Fixed five-hundred-twelve-token splits break tables, legal clauses, and API reference sections. Better defaults include structure-aware splitting on headings, parent-child indexes, and modest overlap on recursive splits.

Embedding model choice affects multilingual EU rollouts. Re-embed when you change models — mixing embeddings in one index destroys recall. Track embedding version in configuration like any dependency.

Hybrid search, reranking, and refusal

Pure vector search misses exact SKUs, ticket IDs, and regulatory article numbers. Hybrid retrieval combines keyword search with dense vectors, then merges scores. A reranker on the top twenty to fifty candidates often lifts precision more than swapping the base LLM.

In regulated Q&A, set a minimum relevance score. If nothing passes, return a safe refusal with a link to human support — never let the model invent an answer when retrieval is empty.

When B2B teams choose RAG first

Security, GDPR, and operations

Treat RAG like internal search plus generation. Lawful basis, retention, and DPIA apply to indexed content and query logs. Encrypt vectors at rest; mirror source-system access in retrieval filters.

Operate with runbooks for ingestion failures, embedding backlog, and spikes in refusal rate. Split metrics: retrieval quality separate from generation quality. Run eval on every index refresh and model upgrade.

From pilot to production

Start with fifty to two hundred golden questions signed off by domain owners. Test access control with real user accounts. Wire refusal and escalation in the UI before marketing calls it “live AI.”

Pilots often reach useful quality in one to three weeks with a focused corpus and clear success metrics. Production hardening — monitoring, cost caps, regression tests — is what separates demos from software.

• RAG use cases
• RAG pipeline engineering

Common misconceptions about RAG

RAG is not “upload PDFs to ChatGPT.” Production systems separate ingestion credentials, index versioning, and user-facing generation. Another myth: bigger chunks always help. Oversized chunks dilute relevance; undersized ones lose legal context. Tune chunk strategy per document type.

RAG also does not eliminate hallucinations — it reduces factual drift when retrieval is good. You still need refusal thresholds, citation UI, and human review on high-risk answers.

Evaluation checklist before go-live

Build a golden set owned by business stakeholders, not only engineering. Include adversarial questions: outdated policy, cross-department access, and ambiguous phrasing. Score retrieval recall and answer faithfulness separately.

Run shadow mode against your current search or ticket macros for two weeks. Compare median time-to-answer and escalation rate — not only “it sounds good.”

Pilot week-by-week

1. 1Week 1: corpus inventory, metadata schema, golden questions v1.
2. 2Week 2: ingestion + first index, retrieval tuning on held-out set.
3. 3Week 3: generation template, citations, refusal UX.
4. 4Week 4: pilot cohort, weekly eval review, backlog for index gaps.

Hybrid architectures with fine-tuning

Mature B2B stacks use RAG for facts and fine-tuning or constrained decoding for format, tone, and routing. The pillar guide on RAG vs fine-tuning walks through decision criteria; cost comparison article models TCO.

Keep adapters versioned and re-run the same golden set when either the index or the model changes — otherwise you cannot tell which layer regressed.