Retrieval-augmented generation

Slide 1 of 30Open this slide

RAG 2.0

Self-Correcting Retrieval for Production

Applied AI Course -- Spring 2026

Week 5

Slide 2 of 30Open this slide

The Evolution of RAG

Stage 1: Naive RAG (2023)

• Single-pass retrieve-then-read
• Fixed top-k chunks concatenated into prompt
• No quality checks on retrieved context
• Works for simple factoid lookup; breaks on complex queries

Stage 2: Advanced RAG (2023-2024)

• Pre-retrieval: query rewriting, HyDE, step-back prompting
• Retrieval: hybrid search, reranking, metadata filtering
• Post-retrieval: context compression, redundancy removal
• Significant gains in retrieval precision

Stage 3: Modular RAG (2024)

• Decomposed pipeline with interchangeable components
• Routing queries to specialized retrieval strategies
• Pluggable retrievers, generators, and evaluators
• Enables experimentation and A/B testing

Stage 4: Agentic RAG (2024-2026)

• LLM-as-orchestrator decides retrieval strategy at runtime
• Self-correcting loops: evaluate, retry, decompose
• Multi-source retrieval with tool use
• Adaptive behavior based on query complexity and retrieval confidence
• By 2026, multi-agent RAG -- specialized agents for decomposition, retrieval, validation, and synthesis in parallel -- is the dominant enterprise production baseline

Sources

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
• Precise Zero-Shot Dense Retrieval without Relevance Labels / HyDE (Gao et al., 2022)
• Take a Step Back: Evoking Reasoning via Abstraction in LLMs (Zheng et al., 2023)
• Corrective Retrieval Augmented Generation / CRAG (Yan et al., ICLR 2024)
• Self-RAG: Learning to Retrieve, Generate, and Critique (Asai et al., ICLR 2024)
• Adaptive-RAG (Jeong et al., NAACL 2024)

Slide 3 of 30Open this slide

Why Naive RAG Fails

Retrieval, not generation, is the failure point. 2026 industry analysis attributes 73% of RAG breakdowns to retrieval -- which reframes where engineering effort belongs: fix retrieval before touching the generator.

Wrong chunks retrieved -- Embedding similarity does not guarantee semantic relevance. A query about "Python GIL performance" may retrieve chunks about "Python snake species" because the embeddings are close in vector space.

Missing context -- Relevant information is split across multiple chunks that never get retrieved together. Chunking destroys document-level reasoning and cross-reference resolution.

Entity blindness -- Dense retrievers struggle with named entities, acronyms, and domain-specific identifiers. "AAPL Q3 2025 revenue" requires exact lexical matching that cosine similarity cannot provide.

Outdated information -- The index reflects a point-in-time snapshot. Without freshness signals or re-indexing pipelines, stale documents silently corrupt answers.

Hallucinated answers -- When retrieval fails silently, the LLM generates plausible-sounding answers from parametric memory rather than admitting uncertainty. The user has no signal that retrieval was inadequate.

No feedback loop -- Naive RAG is open-loop: retrieve once, generate once, return. There is no mechanism to detect failure, retry with a different strategy, or escalate to the user.

Sources

• RAG in Production: Retrieval as the 73% Failure Point (Lush Binary, 2026)

Slide 4 of 30Open this slide

RAG 2.0: Retrieval That Knows When It's Wrong

Core principle -- A production RAG system must be closed-loop. It should detect retrieval failure, diagnose the cause, and apply corrective strategies before generating an answer.

Self-assessment -- The system evaluates whether retrieved documents are relevant, sufficient, and consistent before proceeding to generation. Low-confidence retrievals trigger corrective actions.

Adaptive strategy -- Different queries require different retrieval approaches. Simple factoid queries need keyword precision; complex analytical queries need multi-hop reasoning; ambiguous queries need clarification.

Hybrid retrieval -- Combine dense vector search with sparse keyword search, then fuse and rerank results. Each modality covers the other's weaknesses.

Iterative refinement -- When the first retrieval pass is insufficient, rewrite the query, decompose it, or retrieve from alternative sources. Allow multiple retrieval attempts before generating.

Grounded generation -- Every claim in the output should be traceable to a retrieved source. If a claim cannot be grounded, flag it or suppress it.

This week's goal -- Build the mental model and practical skills to implement self-correcting RAG pipelines that are reliable enough for production deployment.

Slide 5 of 30Open this slide

Vector Search Limitations

Lexical mismatch -- Dense embeddings map semantically similar text to nearby vectors, but they can fail when the surface form matters. Searching for "ERC-20 token standard" may not retrieve a document that only mentions "Ethereum Request for Comments 20" despite being the same concept.

Entity blindness -- Embedding models compress named entities (product IDs, legal case numbers, API endpoint paths) into the same general region of vector space. "CVE-2024-3094" and "CVE-2024-21762" are entirely different vulnerabilities but embed almost identically.

Negation insensitivity -- "Systems that do NOT use Kubernetes" and "Systems that use Kubernetes" produce nearly identical embeddings. Dense retrieval cannot reliably handle negation, exclusion, or contrastive queries.

Numeric reasoning -- Queries like "revenue greater than $50M in Q3" require range filtering and numeric comparison. Embedding similarity has no mechanism for numeric operations.

Rare terminology -- Domain-specific jargon, newly coined terms, and low-frequency tokens are poorly represented in general-purpose embedding models. Out-of-vocabulary terms get subword-tokenized into meaningless fragments.

High recall, low precision -- Vector search casts a wide semantic net. For queries requiring exact matching (error codes, configuration keys, specific identifiers), sparse retrieval is strictly superior.

Slide 6 of 30Open this slide

BM25 and Keyword Search: When Exact Match Matters

BM25 scoring -- Best Match 25 ranks documents using term frequency (TF), inverse document frequency (IDF), and document length normalization. It rewards documents containing rare query terms and penalizes very long documents.

BM25 formula (simplified)

score(q, d) = SUM[ IDF(t) * (TF(t,d) * (k1 + 1)) / (TF(t,d) + k1 * (1 - b + b * |d|/avgdl)) ]

• k1 controls term frequency saturation (typical: 1.2--2.0)
• b controls document length normalization (typical: 0.75)

When BM25 wins over vectors

• Exact entity lookup: product SKUs, error codes, legal citations
• Boolean queries with required terms
• Queries with rare or domain-specific terminology
• Structured metadata search (dates, identifiers, categories)
• Negation and exclusion patterns

Practical considerations

• BM25 requires a traditional inverted index (Elasticsearch, Lucene, SQLite FTS5)
• Tokenization and stemming choices directly affect recall
• No model inference required: sub-millisecond latency at scale
• Works out of the box with no training data
• Cannot handle synonyms, paraphrases, or semantic similarity without expansion

Sources

• The Probabilistic Relevance Framework: BM25 and Beyond (Robertson & Zaragoza, 2009)
• Elasticsearch Documentation
• SQLite FTS5 Extension

Slide 7 of 30Open this slide

Hybrid Search: Vector + BM25 + Reciprocal Rank Fusion

Reciprocal Rank Fusion (RRF)

RRF_score(d) = SUM[ 1 / (k + rank_i(d)) ]   for each ranker i

• k is a constant (typically 60) that dampens the impact of high-ranking outliers
• Documents appearing in multiple result lists get boosted
• No score normalization needed -- works directly on rank positions
• Outperforms linear score combination in practice (Cormack et al., 2009)

Sources

• Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods (Cormack et al., SIGIR 2009)

Slide 8 of 30Open this slide

Cross-Encoder Reranking: Second-Pass Precision

Why two passes?

• Bi-encoders are fast but encode query and document independently -- no cross-attention
• Cross-encoders jointly process query + document through full transformer attention
• Cross-encoders are 100-1000x slower but significantly more accurate
• Typical pipeline: retrieve 100 with bi-encoder, rerank top 20 with cross-encoder
• ColBERT is a middle ground: late interaction compares query and document at the token level (MaxSim) -- bi-encoder-class speed, near cross-encoder accuracy

Now a near-mandatory production stage. A cross-encoder reranker after retrieval and before generation is standard in 2026 RAG. Hosted options like Pinecone Inference and Cohere Rerank fold reranking into the retrieval service, removing the need to self-host a reranker.

Production rerankers

Sources

• ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (Santhanam et al., NAACL 2022)
• BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of IR Models (Thakur et al., NeurIPS 2021)
• Cohere Rerank Documentation
• BGE Reranker Models (BAAI)
• Top Vector Databases for LLM Applications — Reranking as a Standard Stage (Second Talent)

Slide 9 of 30Open this slide

Benchmark Results: Hybrid Outperforms Either Approach Alone

BEIR benchmark results (NDCG@10) across retrieval strategies

Key observations

• Hybrid consistently outperforms both BM25-only and dense-only retrieval across all domains
• BM25 still wins on entity-heavy and exact-match tasks (e.g., EntityQuestions: BM25 71.4 vs Dense 52.1)
• Reranking adds 4-6 points NDCG@10 on average -- the single highest-ROI addition to any RAG pipeline
• The gap widens on out-of-domain data where embedding models have not been fine-tuned
• Hybrid search with reranking achieves 95%+ of fine-tuned retriever performance with zero training data

Practical takeaway -- Start every production RAG system with hybrid search + reranking. It is the highest-impact, lowest-effort improvement available.

Sources

• BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of IR Models (Thakur et al., NeurIPS 2021)
• Text Embeddings by Weakly-Supervised Contrastive Pre-training / E5 (Wang et al., 2022)

Slide 10 of 30Open this slide

Corrective RAG (CRAG)

Paper -- Yan et al., "Corrective Retrieval Augmented Generation" (ICLR 2024)

Core idea -- Embed self-evaluation of retrieved documents before generation. If the evidence is weak, re-query or fall back to web search rather than generating from bad context. This is a 2026 default for high-stakes domains where hallucinations are unacceptable.

Three retrieval confidence levels

• Correct -- At least one retrieved document is highly relevant. Proceed to generation with knowledge refinement to strip irrelevant sentences.
• Incorrect -- All retrieved documents are irrelevant. Discard them entirely and fall back to web search for fresh, relevant context.
• Ambiguous -- Some documents are marginally relevant. Combine refined internal retrieval results with supplementary web search results.

Lightweight retrieval evaluator -- A fine-tuned T5-Large model scores each retrieved document as relevant or irrelevant given the query. This is cheaper than using the main LLM for evaluation.

Knowledge refinement -- Decompose each relevant document into fine-grained knowledge strips (sentence-level). Score each strip independently and discard irrelevant ones. This eliminates noise within documents, not just across documents.

Results -- CRAG improves over standard RAG by 5-10% on PopQA, Biography, and PubQA benchmarks, with the largest gains on queries where naive retrieval returns poor results.

Sources

• Corrective Retrieval Augmented Generation (Yan et al., ICLR 2024)
• RAG in 2026: A Practical Blueprint for Retrieval-Augmented Generation (dev.to)

Slide 11 of 30Open this slide

CRAG Execution Flow

Knowledge refinement (strip-level filtering)

Each sentence is independently evaluated for relevance to the original query. Irrelevant sentences within otherwise-relevant documents are discarded before generation.

Sources

• Corrective Retrieval Augmented Generation (Yan et al., ICLR 2024)

Slide 12 of 30Open this slide

Self-RAG: Learn to Retrieve, Generate, and Self-Reflect

Paper -- Asai et al., "Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection" (ICLR 2024)

Core idea -- Train a single LM to adaptively decide when to retrieve, evaluate the relevance of retrieved passages, and assess whether its own generation is supported by the evidence. All of this is done through special reflection tokens predicted by the model itself.

Key differences from standard RAG

• Retrieval is on-demand, not mandatory -- the model predicts when it needs external knowledge
• Self-evaluation is built into generation via learned critique tokens
• No separate retrieval evaluator model needed -- the LM handles everything
• Inference-time controllability through reflection token thresholds

Training process

1. Train a critic model (GPT-4 distillation) to generate reflection tokens for training data
2. Train the retriever to fetch passages when the model predicts [Retrieve=Yes]
3. Train the generator LM on text interleaved with reflection tokens using standard next-token prediction
4. At inference, the model generates reflection tokens as part of its output and uses them to guide behavior

Results -- Self-RAG (Llama2-7B/13B) outperforms ChatGPT and retrieval-augmented Llama2-chat on open-domain QA, reasoning, and fact verification tasks, while citing fewer but more relevant passages.

Sources

• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (Asai et al., ICLR 2024)

Slide 13 of 30Open this slide

Self-RAG Reflection Tokens

Token          Purpose                        Values
========================================================================
[Retrieve]     Should I retrieve?             { Yes, No, Continue }
[IsRel]        Is passage relevant to query?  { Relevant, Irrelevant }
[IsSup]        Is generation supported?       { Fully, Partially, None }
[IsUse]        Is response useful overall?    { 5, 4, 3, 2, 1 }

Generation flow with reflection tokens

Inference-time control -- By adjusting thresholds on [IsSup] and [IsUse] scores, you can trade off between:

• High precision (strict thresholds): only output claims fully supported by evidence
• High recall (relaxed thresholds): allow partially supported claims for broader coverage
• No retrieval (force [Retrieve=No]): use the model as a standard LM

Sources

• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (Asai et al., ICLR 2024)

Slide 14 of 30Open this slide

Adaptive RAG: Route Queries by Complexity

Paper -- Jeong et al., "Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity" (NAACL 2024)

Classify Query Complexity

A lightweight classifier (trained on silver labels from LLM evaluation) categorizes each incoming query:

• Level A -- Simple factoid questions answerable by a single passage
• Level B -- Moderate questions requiring synthesis across 2-3 sources
• Level C -- Complex multi-hop questions requiring iterative reasoning

Route to Strategy

Execute Retrieval

• Level A: Single retrieval pass, direct generation. Fast and cheap.
• Level B: Retrieve, rerank, verify relevance, generate with self-check.
• Level C: Iterative retrieval with query decomposition. Each sub-question triggers its own retrieval cycle. Results are aggregated before final generation.

Generate with Appropriate Depth

Match generation effort to query complexity. Simple queries get concise answers; complex queries get structured, multi-part responses with citations.

Sources

• Adaptive-RAG: Learning to Adapt Retrieval-Augmented LLMs through Question Complexity (Jeong et al., NAACL 2024)

Slide 15 of 30Open this slide

Query Rewriting: Transform Queries for Better Retrieval

Problem -- User queries are often vague, conversational, or poorly structured for retrieval. The gap between how humans ask questions and how information is indexed is a primary cause of retrieval failure.

Technique 1: LLM-based rewriting

Prompt the LLM to reformulate the query before retrieval. Effective for expanding abbreviations, resolving coreferences, and adding domain context.

Original:  "Why is my app slow after the update?"
Rewritten: "Application performance degradation after software update
            causes and troubleshooting"

Technique 2: HyDE (Hypothetical Document Embeddings)

Generate a hypothetical answer to the query, then embed that answer and use it as the search vector. The hypothesis is closer in embedding space to real documents than the original query.

Query:     "How does mRNA vaccine work?"
HyDE doc:  "mRNA vaccines deliver synthetic messenger RNA encoding a
            viral protein. Cells produce the protein, triggering..."
--> Embed the HyDE doc, not the query

Technique 3: Step-back prompting

Abstract the query to a higher-level concept before retrieval. Retrieve broader context first, then answer the specific question.

Original:  "What happens if I set batch_size=1 in PyTorch DataLoader?"
Step-back: "How does batch size affect PyTorch DataLoader behavior and
            training performance?"

Technique 4: Sub-question decomposition

Break complex queries into atomic sub-questions, retrieve for each independently, and merge context before generating.

Sources

• Precise Zero-Shot Dense Retrieval without Relevance Labels / HyDE (Gao et al., ACL 2023)
• Take a Step Back: Evoking Reasoning via Abstraction in LLMs (Zheng et al., 2023)

Slide 16 of 30Open this slide

Multi-Step Retrieval: Iterative Refinement of Context

Problem -- Complex queries cannot be answered by a single retrieval pass. Multi-hop questions require chaining information across documents, where the answer to one sub-question determines what to retrieve next.

IRCoT (Interleaving Retrieval with Chain-of-Thought)

Alternate between generating reasoning steps and retrieving new evidence. Each reasoning step may surface new entities or concepts that trigger a fresh retrieval.

ITER-RETGEN (Iterative Retrieval-Generation)

Alternate between retrieval and generation for a fixed number of iterations. Each generation provides additional context that improves the next retrieval query.

Practical implementation pattern

1. Retrieve initial context with the original query
2. Generate a draft answer or reasoning chain
3. Extract key entities and claims from the draft
4. Formulate follow-up queries for missing information
5. Retrieve additional context
6. Generate the final answer with the full aggregated context

When to use multi-step -- Multi-hop questions, comparison queries, queries requiring temporal reasoning, and any question where the required context is spread across documents that would not all surface in a single retrieval pass.

Cost consideration -- Each retrieval step adds latency and API cost. Limit iteration depth (typically 2-4 rounds) and use early stopping when confidence is sufficient.

Sources

• Interleaving Retrieval with Chain-of-Thought Reasoning / IRCoT (Trivedi et al., ACL 2023)
• Enhancing Retrieval-Augmented LLMs with Iterative Retrieval-Generation Synergy / ITER-RETGEN (Shao et al., EMNLP 2023)

Slide 17 of 30Open this slide

Agentic RAG: Agents That Plan Retrieval Dynamically

Concept -- Instead of a fixed retrieval pipeline, an LLM agent dynamically plans and executes retrieval strategies based on the query, intermediate results, and available tools.

2026 production baseline -- Multi-agent RAG -- specialized agents for query decomposition, retrieval, validation, and synthesis running in parallel -- is now the dominant enterprise pattern, superseding naive single-retriever pipelines as the default for new builds.

Agent capabilities

• Tool selection -- Choose between vector search, keyword search, SQL queries, API calls, and web search based on query characteristics
• Query planning -- Decompose complex questions into retrieval sub-tasks with dependency ordering
• Result evaluation -- Assess whether retrieved context is sufficient before generating
• Retry with adaptation -- If retrieval fails, reformulate the query or switch to an alternative source
• Aggregation -- Synthesize information from multiple retrieval steps into a coherent answer

Implementation with tool-use LLMs

tools = [
    vector_search(query, collection, top_k),
    bm25_search(query, index, top_k),
    sql_query(query, database),
    web_search(query),
    document_lookup(doc_id, section),
]
# The agent decides which tools to call, in what order,
# and whether to iterate based on intermediate results

LangGraph / LlamaIndex Workflows -- Modern frameworks model agentic RAG as a state machine with conditional edges. Nodes represent retrieval/generation steps; edges represent evaluation-based routing decisions.

Trade-offs -- Agentic RAG is the most capable but also the most expensive and hardest to debug. Each agent decision adds latency, cost, and a potential failure point. Use it when simpler pipelines cannot handle the query complexity.

Sources

• LangChain / LangGraph Documentation
• LlamaIndex Documentation
• Agentic Retrieval-Augmented Generation: A Survey (Singh et al., 2025)

Slide 18 of 30Open this slide

Multi-Source RAG: Combining Heterogeneous Knowledge

Vector Store

Dense retrieval over document embeddings. Best for semantic similarity search over unstructured text -- technical docs, knowledge bases, support articles.

Typical stack: Pinecone, Weaviate, Qdrant, pgvector

Structured Database

Text-to-SQL or text-to-query for structured data. Best for numerical queries, aggregations, filtering, and joins across relational data.

Typical stack: PostgreSQL, BigQuery, Snowflake

API Endpoints

Real-time data from external services. Best for current information: stock prices, weather, user profiles, live system status.

Typical stack: REST/GraphQL APIs with tool-use integration

Web Search

Broad internet search for fresh, general knowledge. Best as a fallback when internal sources lack coverage, or for recent events beyond the training cutoff.

Typical stack: Tavily, Serper, Brave Search API

Knowledge Graph

Entity-relationship traversal for structured reasoning. Best for multi-hop questions involving relationships between entities -- org charts, dependency graphs, ontologies.

Typical stack: Neo4j, Amazon Neptune, property graphs

Code Repositories

AST-aware retrieval over codebases. Best for code search, API usage examples, and understanding implementation details. Requires specialized chunking strategies.

Typical stack: GitHub Search API, Sourcegraph, tree-sitter based indexing

Slide 19 of 30Open this slide

Contextual Compression: Extract Only What Matters

Problem -- Retrieved chunks contain a mix of relevant and irrelevant information. Feeding entire chunks into the LLM wastes context window tokens and introduces noise that can degrade answer quality.

Approach -- After retrieval, apply a compression step that extracts or summarizes only the portions of each document relevant to the specific query.

Technique 1: Extractive compression

An LLM or smaller model identifies and extracts the specific sentences or passages within each retrieved chunk that are relevant to the query. Everything else is discarded.

• Fast with a small classification model
• Preserves original wording (important for citation accuracy)
• Can use sentence-level relevance scoring

Technique 2: Abstractive compression

An LLM summarizes each retrieved chunk into a concise statement that directly addresses the query. The summary replaces the original chunk in the generation context.

• Better information density per token
• Risk of introducing errors in the compression step
• Higher latency due to LLM call per chunk

Technique 3: Contextual chunk headers

Prepend each chunk with metadata context (document title, section heading, date) so the LLM can properly situate the information without reading surrounding chunks.

Impact -- Contextual compression typically reduces input tokens by 50-70% while maintaining or improving answer quality. On long-context queries, this directly translates to lower latency and cost.

Slide 20 of 30Open this slide

Citation and Grounding: Linking Answers Back to Sources

Why grounding matters -- Without citations, RAG output is indistinguishable from hallucination. Users cannot verify claims, audit answers, or build trust in the system. Regulatory environments (finance, healthcare, legal) require traceability.

Inline citation generation

Instruct the LLM to generate citations as part of its output, referencing specific retrieved documents by index or identifier.

The transformer architecture uses multi-head attention to process
input tokens in parallel [Source 2, Section 3.1], achieving
significant speedups over recurrent approaches [Source 5].

Post-hoc attribution

After generation, use an NLI (natural language inference) model to match each generated sentence to supporting passages. Sentences without sufficient support are flagged or removed.

Grounding verification pipeline

1. Split generated answer into individual claims
2. For each claim, score entailment against all retrieved passages
3. Classify: Supported (entailment score > 0.8), Partially supported (0.5-0.8), Unsupported (< 0.5)
4. Surface unsupported claims to the user with appropriate caveats

Production patterns

• Return source documents alongside the answer for user inspection
• Highlight which parts of the source were used
• Provide confidence scores per claim
• Allow users to drill down from a claim to its supporting evidence

Slide 21 of 30Open this slide

Streaming RAG: Real-Time Retrieval in Conversational Systems

Challenge -- Users expect sub-second response times in chat interfaces. Standard RAG adds 500ms-3s of retrieval latency before the first token appears. In multi-turn conversations, this compounds with conversation history management.

Streaming architecture

1. Parallel retrieval + first-token generation -- Begin generating a preamble ("Let me look into that...") while retrieval executes in parallel. Replace with grounded content once results arrive.

2. Speculative retrieval -- Pre-fetch likely-needed context based on conversation trajectory before the user finishes typing. Cache results for immediate use.

3. Incremental context injection -- Start generating with initial results, then inject additional context as reranking and secondary retrievals complete.

Conversation memory for RAG

• Query contextualization -- Rewrite the current query to be self-contained by incorporating relevant conversation history. "What about their revenue?" becomes "What is Nvidia's revenue for Q3 2025?"
• Sliding window -- Keep the last N turns in the retrieval query for context
• Summary buffer -- Maintain a running summary of the conversation for long sessions

Latency budget for production chat RAG

Slide 22 of 30Open this slide

RAG-Fusion: Multiple Query Variants, Merged Results

Paper -- Rackauckas (2023), inspired by Reciprocal Rank Fusion research

Why it works -- Different phrasings of the same question activate different regions of the embedding space and match different keyword patterns. Merging via RRF surfaces documents that are consistently relevant across multiple query interpretations.

Query variant generation prompt

Given this query, generate 3 different search queries that
would help find relevant information. Vary the specificity,
terminology, and angle of each variant.

Trade-off -- N query variants means N retrieval calls. Typically N=3-5 provides the best cost-quality ratio. Beyond that, diminishing returns.

Sources

• RAG-Fusion: A New Take on Retrieval-Augmented Generation (Rackauckas, 2024)
• Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods (Cormack et al., SIGIR 2009)

Slide 23 of 30Open this slide

Production RAG Pipeline Architecture

Key architectural decisions

• Retrieval router decides which backends to query based on query classification
• Response cache (semantic similarity cache) avoids re-processing near-duplicate queries
• Self-eval loop feeds back into retrieval if quality is below threshold
• Grounding check runs in parallel with response streaming where possible
• Every component emits structured logs for observability and debugging

Slide 24 of 30Open this slide

Indexing Pipeline: From Raw Documents to Searchable Index

Document Ingestion

• Ingest from heterogeneous sources: S3, Google Drive, Confluence, Git repos, databases
• Extract text from PDF (PyMuPDF, Unstructured), DOCX, HTML, Markdown
• Preserve document structure: headings, tables, code blocks, lists
• Extract and store metadata: title, author, date, source URL, permissions

Chunking Strategy

• Fixed-size (512 tokens, 256 overlap): simple, predictable, baseline approach
• Recursive character splitting: split on paragraphs, then sentences, then words
• Semantic chunking: use embedding similarity to find natural breakpoints
• Document-aware: respect section boundaries, never split tables or code blocks
• Optimal chunk size depends on embedding model (most peak at 256-512 tokens)

Embedding & Enrichment

• Generate dense embeddings (text-embedding-3-large, BGE-M3, Cohere embed-v3)
• Extract keywords for BM25 index (tokenize, stem, remove stopwords)
• Generate contextual headers: prepend document/section context to each chunk
• Optionally generate summaries or questions-this-chunk-answers for multi-representation indexing

Storage & Indexing

• Write dense vectors to vector store with metadata filters (Qdrant, Weaviate, pgvector)
• Write tokenized text to inverted index (Elasticsearch, OpenSearch, SQLite FTS5)
• Store raw chunks in document store for retrieval-time access
• Build HNSW or IVF-PQ index for approximate nearest neighbor search
• Version the index: track which documents are indexed, enable incremental updates

Sources

• Unstructured Library for Document Parsing
• BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity (Chen et al., 2024)
• Text Embeddings by Weakly-Supervised Contrastive Pre-training / E5 (Wang et al., 2022)

Slide 25 of 30Open this slide

Monitoring RAG: Retrieval Metrics, Answer Quality, Drift Detection

Retrieval quality metrics

• Context relevance -- What fraction of retrieved chunks are actually relevant to the query? Measured by LLM-as-judge or human annotation. Target: >0.7 for top-5 chunks.
• Context recall -- Does the retrieved context contain the information needed to answer? Compare against ground-truth answer. Target: >0.85.
• Retrieval latency -- P50, P95, P99 for the full retrieval pipeline. Budget: <200ms P95.
• Hit rate -- Fraction of queries where at least one relevant document is in top-k. Target: >0.90.

Answer quality metrics

• Faithfulness -- Is the generated answer supported by the retrieved context? Decompose into claims and check entailment. Target: >0.90.
• Answer relevance -- Does the answer address the user's question? LLM-as-judge scoring. Target: >0.85.
• Hallucination rate -- Fraction of generated claims not supported by any retrieved document. Target: <0.05.

Operational monitoring

• Query volume and patterns -- Detect shifts in query distribution that may indicate index staleness
• Embedding drift -- Monitor embedding model output stability; retraining or model updates can silently shift the vector space
• Index freshness -- Track the age of indexed documents; alert when the most recent document is older than threshold
• Feedback loops -- Log user thumbs-up/down, follow-up questions (implicit negative signal), and query reformulations

Frameworks -- Ragas, DeepEval, and LangSmith provide automated evaluation pipelines for RAG systems with LLM-as-judge scoring.

Sources

• RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)
• DeepEval: LLM Evaluation Framework
• LangSmith Observability Platform

Slide 26 of 30Open this slide

Common Failure Modes and Debugging Strategies

Failure: Relevant document exists but is not retrieved

• Check chunking: was the relevant passage split across chunk boundaries?
• Check embedding quality: does the query embed near the target document?
• Try BM25 search directly: is this a lexical-match problem?
• Solution: adjust chunk size/overlap, add hybrid search, try query rewriting

Failure: Retrieved documents are relevant but answer is wrong

• Check context window: are chunks being truncated due to token limits?
• Check ordering: is the most relevant chunk buried among irrelevant ones?
• Solution: add reranking, apply contextual compression, reduce top-k

Failure: LLM ignores retrieved context and hallucinates

• Common with over-parameterized models that have strong priors
• Check prompt: is the instruction to use provided context explicit enough?
• Solution: strengthen grounding instructions, add citation requirements, use smaller specialized models

Failure: System works in testing but fails in production

• Query distribution shift: real users ask questions your test set did not cover
• Data freshness: index was built on a snapshot that is now stale
• Solution: continuous evaluation with production traffic sampling, automated re-indexing

Failure: Latency spikes under load

• Vector search ANN index not warmed, cold-start penalty
• Reranker becomes bottleneck with large candidate sets
• Solution: pre-warm indexes, cap reranking candidate count, add response caching

Debugging toolkit -- Log the full pipeline trace for every query: original query, rewritten query, retrieved doc IDs with scores, reranked order, final prompt, and generated answer. Without this, debugging is guesswork.

Slide 27 of 30Open this slide

Cost Optimization: Caching, Tiered Retrieval, Model Routing

Semantic caching

Cache responses for semantically similar queries, not just exact matches. Embed each query and check cache with a similarity threshold (>0.95 cosine). Cache hit rates of 15-30% are typical for internal knowledge-base applications.

Tiered retrieval

Route queries to the cheapest tier that can handle them. Use a lightweight classifier (or heuristics based on query length and type) to select the tier.

Model routing for generation

• Simple factoid answers: small model (Haiku, GPT-4o-mini) -- fast, cheap
• Complex synthesis: large model (Opus, GPT-4o) -- slower, expensive
• Route based on query complexity classification and retrieval confidence

Embedding cost reduction

• Batch embedding requests (API calls charge per token, not per request)
• Cache embeddings for repeated queries
• Use dimensionality reduction (Matryoshka embeddings) for initial ANN search, full dimensions for reranking

Infrastructure savings

• Use quantized vector indexes (binary or scalar quantization) for 4-8x memory reduction
• Offload cold data to disk-based indexes, keep hot data in memory
• Right-size GPU instances for reranking workloads

Slide 28 of 30Open this slide

Key Takeaways

Hybrid Search Is the Baseline

Never ship vector-only retrieval. Combine dense embeddings with BM25 sparse search and reciprocal rank fusion. Add a cross-encoder reranker for the highest-impact single improvement to any RAG pipeline.

Close the Loop

Open-loop RAG (retrieve once, generate once) fails silently. Implement retrieval evaluation -- even a simple relevance classifier -- and route to corrective actions when confidence is low. CRAG and Self-RAG provide proven patterns.

Match Strategy to Query

Not all queries need the same retrieval depth. Simple factoid lookups should be fast and cheap. Complex multi-hop questions warrant iterative retrieval with query decomposition. Use Adaptive RAG routing to allocate resources proportionally.

Observe Everything

Log the full pipeline trace: query, rewrite, retrieved docs, scores, reranked order, prompt, and output. Without observability, you cannot diagnose failures, measure improvements, or detect drift. Instrument before you optimize.

Ground Every Claim

Every statement in a RAG output should trace back to a retrieved source. Implement citation generation or post-hoc attribution. Flag unsupported claims rather than presenting them as facts. Trust is earned through verifiability.

Optimize the Pipeline, Not Just the Model

Chunking strategy, embedding model selection, reranker choice, and caching architecture often matter more than upgrading the LLM. Profile your pipeline end-to-end before spending on a larger generator.

The Knowledge Base Is the Primary Investment

The 2026 consensus: knowledge-base quality -- chunking, freshness, and deduplication -- matters more for RAG performance than model choice. Treat the knowledge source as the asset you invest in first; a frontier model cannot rescue a stale, noisy, or poorly-chunked corpus.

Sources

• RAG in 2026: Knowledge-Base Quality Over Model Choice (Techment)

Slide 29 of 30Open this slide

Key Papers in Self-Correcting RAG

Recommended reading order -- Start with CRAG and Self-RAG for the core self-correction patterns. Then read Adaptive-RAG for routing. IRCoT for multi-step retrieval. RAG-Fusion for query expansion.

Slide 30 of 30Open this slide

RAG 2.0

Retrieval that knows when it's wrong -- and fixes itself.

Next week: GraphRAG and Global Context — graph-based retrieval and corpus-wide questions.