# Week 5: Retrieval-augmented generation

Applied AI Course — Sawan Ruparel

Public self-study edition. See https://applied-ai.sawanruparel.com/updates/ for version notes.

# RAG 2.0

## Self-Correcting Retrieval for Production

Applied AI Course -- Spring 2026

Week 5

---

# The Evolution of RAG

```mermaid
flowchart LR
    S1["Stage 1<br/>Naive RAG<br/>(2023)"] --> S2["Stage 2<br/>Advanced RAG<br/>(2023-24)"]
    S2 --> S3["Stage 3<br/>Modular RAG<br/>(2024)"]
    S3 --> S4["Stage 4<br/>Agentic RAG<br/>(2024-26)"]
```

## Stage 1: Naive RAG (2023)

- Single-pass retrieve-then-read
- Fixed top-k chunks concatenated into prompt
- No quality checks on retrieved context
- Works for simple factoid lookup; breaks on complex queries

## Stage 2: Advanced RAG (2023-2024)

- Pre-retrieval: query rewriting, HyDE, step-back prompting
- Retrieval: hybrid search, reranking, metadata filtering
- Post-retrieval: context compression, redundancy removal
- Significant gains in retrieval precision

## Stage 3: Modular RAG (2024)

- Decomposed pipeline with interchangeable components
- Routing queries to specialized retrieval strategies
- Pluggable retrievers, generators, and evaluators
- Enables experimentation and A/B testing

## Stage 4: Agentic RAG (2024-2026)

- LLM-as-orchestrator decides retrieval strategy at runtime
- Self-correcting loops: evaluate, retry, decompose
- Multi-source retrieval with tool use
- Adaptive behavior based on query complexity and retrieval confidence
- By 2026, **multi-agent RAG** -- specialized agents for decomposition, retrieval, validation, and synthesis in parallel -- is the dominant enterprise production baseline

## Sources

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)](https://arxiv.org/abs/2005.11401)
- [Precise Zero-Shot Dense Retrieval without Relevance Labels / HyDE (Gao et al., 2022)](https://arxiv.org/abs/2212.10496)
- [Take a Step Back: Evoking Reasoning via Abstraction in LLMs (Zheng et al., 2023)](https://arxiv.org/abs/2310.06117)
- [Corrective Retrieval Augmented Generation / CRAG (Yan et al., ICLR 2024)](https://arxiv.org/abs/2401.15884)
- [Self-RAG: Learning to Retrieve, Generate, and Critique (Asai et al., ICLR 2024)](https://arxiv.org/abs/2310.11511)
- [Adaptive-RAG (Jeong et al., NAACL 2024)](https://arxiv.org/abs/2403.14403)

---

# Why Naive RAG Fails

> **Retrieval, not generation, is the failure point.** 2026 industry analysis attributes **73% of RAG breakdowns to retrieval** -- which reframes where engineering effort belongs: fix retrieval before touching the generator.

**Wrong chunks retrieved** -- Embedding similarity does not guarantee semantic relevance. A query about "Python GIL performance" may retrieve chunks about "Python snake species" because the embeddings are close in vector space.

**Missing context** -- Relevant information is split across multiple chunks that never get retrieved together. Chunking destroys document-level reasoning and cross-reference resolution.

**Entity blindness** -- Dense retrievers struggle with named entities, acronyms, and domain-specific identifiers. "AAPL Q3 2025 revenue" requires exact lexical matching that cosine similarity cannot provide.

**Outdated information** -- The index reflects a point-in-time snapshot. Without freshness signals or re-indexing pipelines, stale documents silently corrupt answers.

**Hallucinated answers** -- When retrieval fails silently, the LLM generates plausible-sounding answers from parametric memory rather than admitting uncertainty. The user has no signal that retrieval was inadequate.

**No feedback loop** -- Naive RAG is open-loop: retrieve once, generate once, return. There is no mechanism to detect failure, retry with a different strategy, or escalate to the user.

## Sources

- [RAG in Production: Retrieval as the 73% Failure Point (Lush Binary, 2026)](https://lushbinary.com/blog/rag-retrieval-augmented-generation-production-guide/)

---

# RAG 2.0: Retrieval That Knows When It's Wrong

**Core principle** -- A production RAG system must be closed-loop. It should detect retrieval failure, diagnose the cause, and apply corrective strategies before generating an answer.

**Self-assessment** -- The system evaluates whether retrieved documents are relevant, sufficient, and consistent before proceeding to generation. Low-confidence retrievals trigger corrective actions.

**Adaptive strategy** -- Different queries require different retrieval approaches. Simple factoid queries need keyword precision; complex analytical queries need multi-hop reasoning; ambiguous queries need clarification.

**Hybrid retrieval** -- Combine dense vector search with sparse keyword search, then fuse and rerank results. Each modality covers the other's weaknesses.

**Iterative refinement** -- When the first retrieval pass is insufficient, rewrite the query, decompose it, or retrieve from alternative sources. Allow multiple retrieval attempts before generating.

**Grounded generation** -- Every claim in the output should be traceable to a retrieved source. If a claim cannot be grounded, flag it or suppress it.

**This week's goal** -- Build the mental model and practical skills to implement self-correcting RAG pipelines that are reliable enough for production deployment.

---

# Vector Search Limitations

**Lexical mismatch** -- Dense embeddings map semantically similar text to nearby vectors, but they can fail when the surface form matters. Searching for "ERC-20 token standard" may not retrieve a document that only mentions "Ethereum Request for Comments 20" despite being the same concept.

**Entity blindness** -- Embedding models compress named entities (product IDs, legal case numbers, API endpoint paths) into the same general region of vector space. "CVE-2024-3094" and "CVE-2024-21762" are entirely different vulnerabilities but embed almost identically.

**Negation insensitivity** -- "Systems that do NOT use Kubernetes" and "Systems that use Kubernetes" produce nearly identical embeddings. Dense retrieval cannot reliably handle negation, exclusion, or contrastive queries.

**Numeric reasoning** -- Queries like "revenue greater than $50M in Q3" require range filtering and numeric comparison. Embedding similarity has no mechanism for numeric operations.

**Rare terminology** -- Domain-specific jargon, newly coined terms, and low-frequency tokens are poorly represented in general-purpose embedding models. Out-of-vocabulary terms get subword-tokenized into meaningless fragments.

**High recall, low precision** -- Vector search casts a wide semantic net. For queries requiring exact matching (error codes, configuration keys, specific identifiers), sparse retrieval is strictly superior.

---

# BM25 and Keyword Search: When Exact Match Matters

**BM25 scoring** -- Best Match 25 ranks documents using term frequency (TF), inverse document frequency (IDF), and document length normalization. It rewards documents containing rare query terms and penalizes very long documents.

**BM25 formula (simplified)**

```
score(q, d) = SUM[ IDF(t) * (TF(t,d) * (k1 + 1)) / (TF(t,d) + k1 * (1 - b + b * |d|/avgdl)) ]
```

- `k1` controls term frequency saturation (typical: 1.2--2.0)
- `b` controls document length normalization (typical: 0.75)

**When BM25 wins over vectors**

- Exact entity lookup: product SKUs, error codes, legal citations
- Boolean queries with required terms
- Queries with rare or domain-specific terminology
- Structured metadata search (dates, identifiers, categories)
- Negation and exclusion patterns

**Practical considerations**

- BM25 requires a traditional inverted index (Elasticsearch, Lucene, SQLite FTS5)
- Tokenization and stemming choices directly affect recall
- No model inference required: sub-millisecond latency at scale
- Works out of the box with no training data
- Cannot handle synonyms, paraphrases, or semantic similarity without expansion

## Sources

- [The Probabilistic Relevance Framework: BM25 and Beyond (Robertson & Zaragoza, 2009)](https://www.staff.city.ac.uk/~sbrp622/papers/foundations_bm25_review.pdf)
- [Elasticsearch Documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html)
- [SQLite FTS5 Extension](https://www.sqlite.org/fts5.html)

---

# Hybrid Search: Vector + BM25 + Reciprocal Rank Fusion

```mermaid
flowchart TB
    Q[Query] --> Emb[Embedding model]
    Q --> Tok[Tokenizer + stemmer]
    Emb --> VIdx["Vector index<br/>(HNSW/IVF)"]
    Tok --> IIdx["Inverted index<br/>(BM25)"]
    VIdx --> Dense[Top-K dense results]
    IIdx --> Sparse[Top-K sparse results]
    Dense --> RRF[Reciprocal rank fusion]
    Sparse --> RRF
    RRF --> Merged[Merged & re-scored list]
```

**Reciprocal Rank Fusion (RRF)**

```
RRF_score(d) = SUM[ 1 / (k + rank_i(d)) ]   for each ranker i
```

- `k` is a constant (typically 60) that dampens the impact of high-ranking outliers
- Documents appearing in multiple result lists get boosted
- No score normalization needed -- works directly on rank positions
- Outperforms linear score combination in practice (Cormack et al., 2009)

## Sources

- [Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods (Cormack et al., SIGIR 2009)](https://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf)

---

# Cross-Encoder Reranking: Second-Pass Precision

```mermaid
flowchart LR
    Q[Query] --> Bi["Bi-encoder<br/>(independent embed)"]
    D[(Doc corpus)] --> Bi
    Bi --> Top100["Top 100 candidates<br/>cosine, ~1ms per 1M"]
    Top100 --> Cross["Cross-encoder<br/>(query + doc, joint attention)<br/>~50ms per pair"]
    Cross --> Top5[Top 5 reranked]
```

**Why two passes?**

- Bi-encoders are fast but encode query and document independently -- no cross-attention
- Cross-encoders jointly process query + document through full transformer attention
- Cross-encoders are 100-1000x slower but significantly more accurate
- Typical pipeline: retrieve 100 with bi-encoder, rerank top 20 with cross-encoder
- **ColBERT** is a middle ground: *late interaction* compares query and document at the token level (MaxSim) -- bi-encoder-class speed, near cross-encoder accuracy

> **Now a near-mandatory production stage.** A cross-encoder reranker after retrieval and before generation is standard in 2026 RAG. Hosted options like Pinecone Inference and Cohere Rerank fold reranking into the retrieval service, removing the need to self-host a reranker.

**Production rerankers**

| Model | Parameters | NDCG@10 (BEIR) | Latency (20 docs) |
|-------|-----------|-----------------|-------------------|
| Cohere Rerank v3 | Proprietary | 59.2 | ~120ms (API) |
| bge-reranker-v2-m3 | 568M | 57.4 | ~80ms (GPU) |
| ms-marco-MiniLM-L-12 | 33M | 49.1 | ~15ms (GPU) |
| ColBERT v2 (late interaction) | 110M | 54.6 | ~5ms (GPU) |

## Sources

- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (Santhanam et al., NAACL 2022)](https://arxiv.org/abs/2112.01488)
- [BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of IR Models (Thakur et al., NeurIPS 2021)](https://arxiv.org/abs/2104.08663)
- [Cohere Rerank Documentation](https://docs.cohere.com/docs/rerank-2)
- [BGE Reranker Models (BAAI)](https://huggingface.co/BAAI/bge-reranker-v2-m3)
- [Top Vector Databases for LLM Applications — Reranking as a Standard Stage (Second Talent)](https://www.secondtalent.com/resources/top-vector-databases-for-llm-applications/)

---

# Benchmark Results: Hybrid Outperforms Either Approach Alone

**BEIR benchmark results (NDCG@10) across retrieval strategies**

| Dataset | BM25 | Dense (E5-Large) | Hybrid (RRF) | Hybrid + Rerank |
|---------|------|-------------------|---------------|-----------------|
| NQ (open-domain QA) | 30.6 | 52.4 | 54.1 | 58.3 |
| HotpotQA (multi-hop) | 60.3 | 54.6 | 65.8 | 69.2 |
| FiQA (financial) | 23.6 | 37.2 | 39.8 | 44.1 |
| SciFact (scientific) | 66.5 | 64.8 | 71.3 | 74.7 |
| TREC-COVID (biomedical) | 59.5 | 65.4 | 71.0 | 75.8 |
| NFCorpus (nutrition) | 32.2 | 34.1 | 36.9 | 38.5 |
| Avg across 18 BEIR tasks | 42.8 | 48.3 | 52.1 | 56.4 |

**Key observations**

- Hybrid consistently outperforms both BM25-only and dense-only retrieval across all domains
- BM25 still wins on entity-heavy and exact-match tasks (e.g., EntityQuestions: BM25 71.4 vs Dense 52.1)
- Reranking adds 4-6 points NDCG@10 on average -- the single highest-ROI addition to any RAG pipeline
- The gap widens on out-of-domain data where embedding models have not been fine-tuned
- Hybrid search with reranking achieves 95%+ of fine-tuned retriever performance with zero training data

**Practical takeaway** -- Start every production RAG system with hybrid search + reranking. It is the highest-impact, lowest-effort improvement available.

## Sources

- [BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of IR Models (Thakur et al., NeurIPS 2021)](https://arxiv.org/abs/2104.08663)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training / E5 (Wang et al., 2022)](https://arxiv.org/abs/2212.03533)

---

# Corrective RAG (CRAG)

**Paper** -- Yan et al., "Corrective Retrieval Augmented Generation" (ICLR 2024)

**Core idea** -- Embed self-evaluation of retrieved documents *before* generation. If the evidence is weak, re-query or fall back to web search rather than generating from bad context. This is a 2026 default for high-stakes domains where hallucinations are unacceptable.

**Three retrieval confidence levels**

```mermaid
flowchart LR
    Eval[Retrieval evaluator] --> Conf{Confidence}
    Conf -->|Correct| Refine[Refine + generate]
    Conf -->|Ambiguous| Mix[Refine + web search]
    Conf -->|Incorrect| Web[Web search fallback]
```

- **Correct** -- At least one retrieved document is highly relevant. Proceed to generation with knowledge refinement to strip irrelevant sentences.
- **Incorrect** -- All retrieved documents are irrelevant. Discard them entirely and fall back to web search for fresh, relevant context.
- **Ambiguous** -- Some documents are marginally relevant. Combine refined internal retrieval results with supplementary web search results.

**Lightweight retrieval evaluator** -- A fine-tuned T5-Large model scores each retrieved document as relevant or irrelevant given the query. This is cheaper than using the main LLM for evaluation.

**Knowledge refinement** -- Decompose each relevant document into fine-grained knowledge strips (sentence-level). Score each strip independently and discard irrelevant ones. This eliminates noise within documents, not just across documents.

**Results** -- CRAG improves over standard RAG by 5-10% on PopQA, Biography, and PubQA benchmarks, with the largest gains on queries where naive retrieval returns poor results.

## Sources

- [Corrective Retrieval Augmented Generation (Yan et al., ICLR 2024)](https://arxiv.org/abs/2401.15884)
- [RAG in 2026: A Practical Blueprint for Retrieval-Augmented Generation (dev.to)](https://dev.to/suraj_khaitan_f893c243958/-rag-in-2026-a-practical-blueprint-for-retrieval-augmented-generation-16pp)

---

# CRAG Execution Flow

```mermaid
flowchart TB
    Q[User query] --> Retrieve[Retrieve top-K docs]
    Retrieve --> Eval["Retrieval evaluator<br/>(T5-Large)"]
    Eval --> Conf{Confidence?}
    Conf -->|Correct| Refine[Refine docs<br/>strip irrelevant sentences]
    Conf -->|Ambiguous| Both[Refine + web search<br/>supplement]
    Conf -->|Incorrect| Discard[Discard all retrieved]
    Discard --> Web[Web search fallback]
    Refine --> Gen[Generate answer]
    Both --> Gen
    Web --> Gen
```

**Knowledge refinement (strip-level filtering)**

```mermaid
flowchart LR
    Doc[Document] --> Split[Split into sentences]
    Split --> Score[Score each sentence]
    Score --> Keep[Keep relevant only]
    Keep --> Concat[Concatenate]
```

Each sentence is independently evaluated for relevance to the original query. Irrelevant sentences within otherwise-relevant documents are discarded before generation.

## Sources

- [Corrective Retrieval Augmented Generation (Yan et al., ICLR 2024)](https://arxiv.org/abs/2401.15884)

---

# Self-RAG: Learn to Retrieve, Generate, and Self-Reflect

**Paper** -- Asai et al., "Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection" (ICLR 2024)

**Core idea** -- Train a single LM to adaptively decide when to retrieve, evaluate the relevance of retrieved passages, and assess whether its own generation is supported by the evidence. All of this is done through special reflection tokens predicted by the model itself.

**Key differences from standard RAG**

- Retrieval is on-demand, not mandatory -- the model predicts when it needs external knowledge
- Self-evaluation is built into generation via learned critique tokens
- No separate retrieval evaluator model needed -- the LM handles everything
- Inference-time controllability through reflection token thresholds

**Training process**

1. Train a critic model (GPT-4 distillation) to generate reflection tokens for training data
2. Train the retriever to fetch passages when the model predicts [Retrieve=Yes]
3. Train the generator LM on text interleaved with reflection tokens using standard next-token prediction
4. At inference, the model generates reflection tokens as part of its output and uses them to guide behavior

**Results** -- Self-RAG (Llama2-7B/13B) outperforms ChatGPT and retrieval-augmented Llama2-chat on open-domain QA, reasoning, and fact verification tasks, while citing fewer but more relevant passages.

## Sources

- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (Asai et al., ICLR 2024)](https://arxiv.org/abs/2310.11511)

---

# Self-RAG Reflection Tokens

```
Token          Purpose                        Values
========================================================================
[Retrieve]     Should I retrieve?             { Yes, No, Continue }
[IsRel]        Is passage relevant to query?  { Relevant, Irrelevant }
[IsSup]        Is generation supported?       { Fully, Partially, None }
[IsUse]        Is response useful overall?    { 5, 4, 3, 2, 1 }
```

**Generation flow with reflection tokens**

```mermaid
flowchart TB
    In["Input:<br/>What causes aurora borealis?"] --> R{"[Retrieve]?"}
    R -->|Yes| Fetch[Retrieve passage<br/>about solar wind]
    R -->|No| Direct[Generate from<br/>parametric memory]
    Fetch --> Rel{"[IsRel]?"}
    Rel -->|Relevant| Gen[Generate response segment]
    Rel -->|Irrelevant| Skip[Skip passage]
    Gen --> Sup{"[IsSup]?"}
    Sup -->|Fully| Use["[IsUse] = 5"]
    Sup -->|Partial| Use
    Use --> Next[Continue or retrieve<br/>for next segment]
```

**Inference-time control** -- By adjusting thresholds on [IsSup] and [IsUse] scores, you can trade off between:

- **High precision** (strict thresholds): only output claims fully supported by evidence
- **High recall** (relaxed thresholds): allow partially supported claims for broader coverage
- **No retrieval** (force [Retrieve=No]): use the model as a standard LM

## Sources

- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (Asai et al., ICLR 2024)](https://arxiv.org/abs/2310.11511)

---

# Adaptive RAG: Route Queries by Complexity

**Paper** -- Jeong et al., "Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity" (NAACL 2024)

```mermaid
flowchart TB
    Q[Incoming query] --> Class[Complexity classifier]
    Class -->|A: simple factoid| A["Single-step retrieval<br/>+ direct generation"]
    Class -->|B: moderate| B["Retrieve + rerank<br/>+ verify"]
    Class -->|C: complex| C["Iterative RAG<br/>+ decomposition"]
    A & B & C --> Ans[Answer]
```

## Classify Query Complexity

A lightweight classifier (trained on silver labels from LLM evaluation) categorizes each incoming query:

- **Level A** -- Simple factoid questions answerable by a single passage
- **Level B** -- Moderate questions requiring synthesis across 2-3 sources
- **Level C** -- Complex multi-hop questions requiring iterative reasoning

## Route to Strategy

| Complexity | Strategy | Example |
|-----------|----------|---------|
| A (Simple) | Single-step retrieval | "When was Python 3.12 released?" |
| B (Moderate) | Retrieve + rerank + verify | "How does Python's GIL affect async performance?" |
| C (Complex) | Multi-step iterative RAG | "Compare Python and Rust memory safety approaches and their impact on web framework performance" |

## Execute Retrieval

- **Level A**: Single retrieval pass, direct generation. Fast and cheap.
- **Level B**: Retrieve, rerank, verify relevance, generate with self-check.
- **Level C**: Iterative retrieval with query decomposition. Each sub-question triggers its own retrieval cycle. Results are aggregated before final generation.

## Generate with Appropriate Depth

Match generation effort to query complexity. Simple queries get concise answers; complex queries get structured, multi-part responses with citations.

## Sources

- [Adaptive-RAG: Learning to Adapt Retrieval-Augmented LLMs through Question Complexity (Jeong et al., NAACL 2024)](https://arxiv.org/abs/2403.14403)

---

# Query Rewriting: Transform Queries for Better Retrieval

**Problem** -- User queries are often vague, conversational, or poorly structured for retrieval. The gap between how humans ask questions and how information is indexed is a primary cause of retrieval failure.

```mermaid
flowchart LR
    Q[Raw query] --> Choose{Rewrite strategy}
    Choose --> Rewrite[LLM rewrite<br/>clarify + expand]
    Choose --> HyDE[HyDE<br/>generate hypothetical doc]
    Choose --> StepBack[Step-back<br/>abstract concept]
    Choose --> Decomp[Decompose<br/>sub-questions]
    Rewrite & HyDE & StepBack & Decomp --> Retrieve[Retrieval]
```

**Technique 1: LLM-based rewriting**

Prompt the LLM to reformulate the query before retrieval. Effective for expanding abbreviations, resolving coreferences, and adding domain context.

```
Original:  "Why is my app slow after the update?"
Rewritten: "Application performance degradation after software update
            causes and troubleshooting"
```

**Technique 2: HyDE (Hypothetical Document Embeddings)**

Generate a hypothetical answer to the query, then embed that answer and use it as the search vector. The hypothesis is closer in embedding space to real documents than the original query.

```
Query:     "How does mRNA vaccine work?"
HyDE doc:  "mRNA vaccines deliver synthetic messenger RNA encoding a
            viral protein. Cells produce the protein, triggering..."
--> Embed the HyDE doc, not the query
```

**Technique 3: Step-back prompting**

Abstract the query to a higher-level concept before retrieval. Retrieve broader context first, then answer the specific question.

```
Original:  "What happens if I set batch_size=1 in PyTorch DataLoader?"
Step-back: "How does batch size affect PyTorch DataLoader behavior and
            training performance?"
```

**Technique 4: Sub-question decomposition**

Break complex queries into atomic sub-questions, retrieve for each independently, and merge context before generating.

## Sources

- [Precise Zero-Shot Dense Retrieval without Relevance Labels / HyDE (Gao et al., ACL 2023)](https://arxiv.org/abs/2212.10496)
- [Take a Step Back: Evoking Reasoning via Abstraction in LLMs (Zheng et al., 2023)](https://arxiv.org/abs/2310.06117)

---

# Multi-Step Retrieval: Iterative Refinement of Context

**Problem** -- Complex queries cannot be answered by a single retrieval pass. Multi-hop questions require chaining information across documents, where the answer to one sub-question determines what to retrieve next.

**IRCoT (Interleaving Retrieval with Chain-of-Thought)**

Alternate between generating reasoning steps and retrieving new evidence. Each reasoning step may surface new entities or concepts that trigger a fresh retrieval.

```mermaid
flowchart LR
    Q["Did the director of Inception<br/>also direct a Batman movie?"] --> R1[Retrieve]
    R1 --> F1["Inception directed by<br/>Christopher Nolan"]
    F1 --> Reason["Reason: check if<br/>Nolan directed Batman"]
    Reason --> R2[Retrieve]
    R2 --> F2["Nolan directed<br/>Dark Knight trilogy"]
    F2 --> Ans["Yes, Christopher Nolan<br/>directed both"]
```

**ITER-RETGEN (Iterative Retrieval-Generation)**

Alternate between retrieval and generation for a fixed number of iterations. Each generation provides additional context that improves the next retrieval query.

**Practical implementation pattern**

1. Retrieve initial context with the original query
2. Generate a draft answer or reasoning chain
3. Extract key entities and claims from the draft
4. Formulate follow-up queries for missing information
5. Retrieve additional context
6. Generate the final answer with the full aggregated context

**When to use multi-step** -- Multi-hop questions, comparison queries, queries requiring temporal reasoning, and any question where the required context is spread across documents that would not all surface in a single retrieval pass.

**Cost consideration** -- Each retrieval step adds latency and API cost. Limit iteration depth (typically 2-4 rounds) and use early stopping when confidence is sufficient.

## Sources

- [Interleaving Retrieval with Chain-of-Thought Reasoning / IRCoT (Trivedi et al., ACL 2023)](https://arxiv.org/abs/2212.10509)
- [Enhancing Retrieval-Augmented LLMs with Iterative Retrieval-Generation Synergy / ITER-RETGEN (Shao et al., EMNLP 2023)](https://arxiv.org/abs/2305.15294)

---

# Agentic RAG: Agents That Plan Retrieval Dynamically

**Concept** -- Instead of a fixed retrieval pipeline, an LLM agent dynamically plans and executes retrieval strategies based on the query, intermediate results, and available tools.

**2026 production baseline** -- Multi-agent RAG -- specialized agents for query decomposition, retrieval, validation, and synthesis running in parallel -- is now the dominant enterprise pattern, superseding naive single-retriever pipelines as the default for new builds.

```mermaid
flowchart TB
    Q[Query] --> Agent[LLM agent]
    Agent --> Plan{Select tool}
    Plan --> VS[vector_search]
    Plan --> BM[bm25_search]
    Plan --> SQL[sql_query]
    Plan --> Web[web_search]
    VS & BM & SQL & Web --> Eval{Sufficient context?}
    Eval -->|no| Agent
    Eval -->|yes| Gen[Generate answer]
```

**Agent capabilities**

- **Tool selection** -- Choose between vector search, keyword search, SQL queries, API calls, and web search based on query characteristics
- **Query planning** -- Decompose complex questions into retrieval sub-tasks with dependency ordering
- **Result evaluation** -- Assess whether retrieved context is sufficient before generating
- **Retry with adaptation** -- If retrieval fails, reformulate the query or switch to an alternative source
- **Aggregation** -- Synthesize information from multiple retrieval steps into a coherent answer

**Implementation with tool-use LLMs**

```python
tools = [
    vector_search(query, collection, top_k),
    bm25_search(query, index, top_k),
    sql_query(query, database),
    web_search(query),
    document_lookup(doc_id, section),
]
# The agent decides which tools to call, in what order,
# and whether to iterate based on intermediate results
```

**LangGraph / LlamaIndex Workflows** -- Modern frameworks model agentic RAG as a state machine with conditional edges. Nodes represent retrieval/generation steps; edges represent evaluation-based routing decisions.

**Trade-offs** -- Agentic RAG is the most capable but also the most expensive and hardest to debug. Each agent decision adds latency, cost, and a potential failure point. Use it when simpler pipelines cannot handle the query complexity.

## Sources

- [LangChain / LangGraph Documentation](https://docs.langchain.com)
- [LlamaIndex Documentation](https://docs.llamaindex.ai)
- [Agentic Retrieval-Augmented Generation: A Survey (Singh et al., 2025)](https://arxiv.org/html/2501.09136v4)

---

# Multi-Source RAG: Combining Heterogeneous Knowledge

## Vector Store

Dense retrieval over document embeddings. Best for semantic similarity search over unstructured text -- technical docs, knowledge bases, support articles.

Typical stack: Pinecone, Weaviate, Qdrant, pgvector

## Structured Database

Text-to-SQL or text-to-query for structured data. Best for numerical queries, aggregations, filtering, and joins across relational data.

Typical stack: PostgreSQL, BigQuery, Snowflake

## API Endpoints

Real-time data from external services. Best for current information: stock prices, weather, user profiles, live system status.

Typical stack: REST/GraphQL APIs with tool-use integration

## Web Search

Broad internet search for fresh, general knowledge. Best as a fallback when internal sources lack coverage, or for recent events beyond the training cutoff.

Typical stack: Tavily, Serper, Brave Search API

## Knowledge Graph

Entity-relationship traversal for structured reasoning. Best for multi-hop questions involving relationships between entities -- org charts, dependency graphs, ontologies.

Typical stack: Neo4j, Amazon Neptune, property graphs

## Code Repositories

AST-aware retrieval over codebases. Best for code search, API usage examples, and understanding implementation details. Requires specialized chunking strategies.

Typical stack: GitHub Search API, Sourcegraph, tree-sitter based indexing

---

# Contextual Compression: Extract Only What Matters

**Problem** -- Retrieved chunks contain a mix of relevant and irrelevant information. Feeding entire chunks into the LLM wastes context window tokens and introduces noise that can degrade answer quality.

**Approach** -- After retrieval, apply a compression step that extracts or summarizes only the portions of each document relevant to the specific query.

**Technique 1: Extractive compression**

An LLM or smaller model identifies and extracts the specific sentences or passages within each retrieved chunk that are relevant to the query. Everything else is discarded.

- Fast with a small classification model
- Preserves original wording (important for citation accuracy)
- Can use sentence-level relevance scoring

**Technique 2: Abstractive compression**

An LLM summarizes each retrieved chunk into a concise statement that directly addresses the query. The summary replaces the original chunk in the generation context.

- Better information density per token
- Risk of introducing errors in the compression step
- Higher latency due to LLM call per chunk

**Technique 3: Contextual chunk headers**

Prepend each chunk with metadata context (document title, section heading, date) so the LLM can properly situate the information without reading surrounding chunks.

**Impact** -- Contextual compression typically reduces input tokens by 50-70% while maintaining or improving answer quality. On long-context queries, this directly translates to lower latency and cost.

---

# Citation and Grounding: Linking Answers Back to Sources

**Why grounding matters** -- Without citations, RAG output is indistinguishable from hallucination. Users cannot verify claims, audit answers, or build trust in the system. Regulatory environments (finance, healthcare, legal) require traceability.

**Inline citation generation**

Instruct the LLM to generate citations as part of its output, referencing specific retrieved documents by index or identifier.

```
The transformer architecture uses multi-head attention to process
input tokens in parallel [Source 2, Section 3.1], achieving
significant speedups over recurrent approaches [Source 5].
```

**Post-hoc attribution**

After generation, use an NLI (natural language inference) model to match each generated sentence to supporting passages. Sentences without sufficient support are flagged or removed.

**Grounding verification pipeline**

1. Split generated answer into individual claims
2. For each claim, score entailment against all retrieved passages
3. Classify: **Supported** (entailment score > 0.8), **Partially supported** (0.5-0.8), **Unsupported** (< 0.5)
4. Surface unsupported claims to the user with appropriate caveats

**Production patterns**

- Return source documents alongside the answer for user inspection
- Highlight which parts of the source were used
- Provide confidence scores per claim
- Allow users to drill down from a claim to its supporting evidence

---

# Streaming RAG: Real-Time Retrieval in Conversational Systems

**Challenge** -- Users expect sub-second response times in chat interfaces. Standard RAG adds 500ms-3s of retrieval latency before the first token appears. In multi-turn conversations, this compounds with conversation history management.

**Streaming architecture**

```mermaid
sequenceDiagram
    autonumber
    participant U as User
    participant App as App
    participant R as Retriever
    participant L as LLM
    U->>App: Query
    par parallel
      App->>R: Retrieve
    and
      App->>L: Stream preamble
    end
    R-->>App: Top-k chunks
    App->>L: Inject grounded context
    L-->>U: Stream grounded answer
```

1. **Parallel retrieval + first-token generation** -- Begin generating a preamble ("Let me look into that...") while retrieval executes in parallel. Replace with grounded content once results arrive.

2. **Speculative retrieval** -- Pre-fetch likely-needed context based on conversation trajectory before the user finishes typing. Cache results for immediate use.

3. **Incremental context injection** -- Start generating with initial results, then inject additional context as reranking and secondary retrievals complete.

**Conversation memory for RAG**

- **Query contextualization** -- Rewrite the current query to be self-contained by incorporating relevant conversation history. "What about their revenue?" becomes "What is Nvidia's revenue for Q3 2025?"
- **Sliding window** -- Keep the last N turns in the retrieval query for context
- **Summary buffer** -- Maintain a running summary of the conversation for long sessions

**Latency budget for production chat RAG**

| Component | Target | Technique |
|-----------|--------|-----------|
| Embedding | <50ms | Cached models, batching |
| Vector search | <30ms | ANN index, pre-filtering |
| BM25 search | <10ms | Inverted index in memory |
| Reranking | <100ms | Distilled cross-encoder |
| LLM first token | <300ms | Streaming, speculative decode |
| **Total TTFT** | **<500ms** | **Parallel execution** |

---

# RAG-Fusion: Multiple Query Variants, Merged Results

**Paper** -- Rackauckas (2023), inspired by Reciprocal Rank Fusion research

```mermaid
flowchart TB
    Q[User query] --> LLM[LLM: generate N variants]
    LLM --> V1["Variant 1<br/>(original phrasing)"]
    LLM --> V2["Variant 2<br/>(technical rewrite)"]
    LLM --> V3["Variant 3<br/>(broader context)"]
    V1 --> R1[Retrieve top-k]
    V2 --> R2[Retrieve top-k]
    V3 --> R3[Retrieve top-k]
    R1 & R2 & R3 --> RRF[Reciprocal rank fusion<br/>across result sets]
    RRF --> Merged[Re-ranked merged results]
    Merged --> Gen[Generate answer]
```

**Why it works** -- Different phrasings of the same question activate different regions of the embedding space and match different keyword patterns. Merging via RRF surfaces documents that are consistently relevant across multiple query interpretations.

**Query variant generation prompt**

```
Given this query, generate 3 different search queries that
would help find relevant information. Vary the specificity,
terminology, and angle of each variant.
```

**Trade-off** -- N query variants means N retrieval calls. Typically N=3-5 provides the best cost-quality ratio. Beyond that, diminishing returns.

## Sources

- [RAG-Fusion: A New Take on Retrieval-Augmented Generation (Rackauckas, 2024)](https://arxiv.org/abs/2402.03367)
- [Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods (Cormack et al., SIGIR 2009)](https://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf)

---

# Production RAG Pipeline Architecture

```mermaid
flowchart TB
    Client[Client request] --> Gateway[API gateway<br/>auth + rate limit]
    Gateway --> QP[Query processor]
    QP --> Router[Retrieval router]
    Router --> V[Vector store]
    Router --> K[BM25 index]
    Router --> S[SQL / API]
    V & K & S --> Fuse[Rank fusion + reranker]
    Fuse --> Gen[Generator LLM]
    Gen --> Eval{Self-eval<br/>CRAG loop}
    Eval -->|low quality| Router
    Eval -->|ok| Ground[Grounding check]
    Ground --> Cache[Response cache]
    Cache --> Out[Response to client]
    Out --> Mon[Monitoring & logging]
```

**Key architectural decisions**

- **Retrieval router** decides which backends to query based on query classification
- **Response cache** (semantic similarity cache) avoids re-processing near-duplicate queries
- **Self-eval loop** feeds back into retrieval if quality is below threshold
- **Grounding check** runs in parallel with response streaming where possible
- **Every component emits structured logs** for observability and debugging

---

# Indexing Pipeline: From Raw Documents to Searchable Index

## Document Ingestion

- Ingest from heterogeneous sources: S3, Google Drive, Confluence, Git repos, databases
- Extract text from PDF (PyMuPDF, Unstructured), DOCX, HTML, Markdown
- Preserve document structure: headings, tables, code blocks, lists
- Extract and store metadata: title, author, date, source URL, permissions

## Chunking Strategy

- **Fixed-size** (512 tokens, 256 overlap): simple, predictable, baseline approach
- **Recursive character splitting**: split on paragraphs, then sentences, then words
- **Semantic chunking**: use embedding similarity to find natural breakpoints
- **Document-aware**: respect section boundaries, never split tables or code blocks
- Optimal chunk size depends on embedding model (most peak at 256-512 tokens)

## Embedding & Enrichment

- Generate dense embeddings (text-embedding-3-large, BGE-M3, Cohere embed-v3)
- Extract keywords for BM25 index (tokenize, stem, remove stopwords)
- Generate contextual headers: prepend document/section context to each chunk
- Optionally generate summaries or questions-this-chunk-answers for multi-representation indexing

## Storage & Indexing

- Write dense vectors to vector store with metadata filters (Qdrant, Weaviate, pgvector)
- Write tokenized text to inverted index (Elasticsearch, OpenSearch, SQLite FTS5)
- Store raw chunks in document store for retrieval-time access
- Build HNSW or IVF-PQ index for approximate nearest neighbor search
- Version the index: track which documents are indexed, enable incremental updates

## Sources

- [Unstructured Library for Document Parsing](https://github.com/Unstructured-IO/unstructured)
- [BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity (Chen et al., 2024)](https://arxiv.org/abs/2402.03216)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training / E5 (Wang et al., 2022)](https://arxiv.org/abs/2212.03533)

---

# Monitoring RAG: Retrieval Metrics, Answer Quality, Drift Detection

**Retrieval quality metrics**

- **Context relevance** -- What fraction of retrieved chunks are actually relevant to the query? Measured by LLM-as-judge or human annotation. Target: >0.7 for top-5 chunks.
- **Context recall** -- Does the retrieved context contain the information needed to answer? Compare against ground-truth answer. Target: >0.85.
- **Retrieval latency** -- P50, P95, P99 for the full retrieval pipeline. Budget: <200ms P95.
- **Hit rate** -- Fraction of queries where at least one relevant document is in top-k. Target: >0.90.

**Answer quality metrics**

- **Faithfulness** -- Is the generated answer supported by the retrieved context? Decompose into claims and check entailment. Target: >0.90.
- **Answer relevance** -- Does the answer address the user's question? LLM-as-judge scoring. Target: >0.85.
- **Hallucination rate** -- Fraction of generated claims not supported by any retrieved document. Target: <0.05.

**Operational monitoring**

- **Query volume and patterns** -- Detect shifts in query distribution that may indicate index staleness
- **Embedding drift** -- Monitor embedding model output stability; retraining or model updates can silently shift the vector space
- **Index freshness** -- Track the age of indexed documents; alert when the most recent document is older than threshold
- **Feedback loops** -- Log user thumbs-up/down, follow-up questions (implicit negative signal), and query reformulations

**Frameworks** -- Ragas, DeepEval, and LangSmith provide automated evaluation pipelines for RAG systems with LLM-as-judge scoring.

## Sources

- [RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)](https://arxiv.org/abs/2309.15217)
- [DeepEval: LLM Evaluation Framework](https://github.com/confident-ai/deepeval)
- [LangSmith Observability Platform](https://docs.langchain.com/langsmith/observability)

---

# Common Failure Modes and Debugging Strategies

**Failure: Relevant document exists but is not retrieved**

- Check chunking: was the relevant passage split across chunk boundaries?
- Check embedding quality: does the query embed near the target document?
- Try BM25 search directly: is this a lexical-match problem?
- Solution: adjust chunk size/overlap, add hybrid search, try query rewriting

**Failure: Retrieved documents are relevant but answer is wrong**

- Check context window: are chunks being truncated due to token limits?
- Check ordering: is the most relevant chunk buried among irrelevant ones?
- Solution: add reranking, apply contextual compression, reduce top-k

**Failure: LLM ignores retrieved context and hallucinates**

- Common with over-parameterized models that have strong priors
- Check prompt: is the instruction to use provided context explicit enough?
- Solution: strengthen grounding instructions, add citation requirements, use smaller specialized models

**Failure: System works in testing but fails in production**

- Query distribution shift: real users ask questions your test set did not cover
- Data freshness: index was built on a snapshot that is now stale
- Solution: continuous evaluation with production traffic sampling, automated re-indexing

**Failure: Latency spikes under load**

- Vector search ANN index not warmed, cold-start penalty
- Reranker becomes bottleneck with large candidate sets
- Solution: pre-warm indexes, cap reranking candidate count, add response caching

**Debugging toolkit** -- Log the full pipeline trace for every query: original query, rewritten query, retrieved doc IDs with scores, reranked order, final prompt, and generated answer. Without this, debugging is guesswork.

---

# Cost Optimization: Caching, Tiered Retrieval, Model Routing

**Semantic caching**

Cache responses for semantically similar queries, not just exact matches. Embed each query and check cache with a similarity threshold (>0.95 cosine). Cache hit rates of 15-30% are typical for internal knowledge-base applications.

**Tiered retrieval**

| Tier | Latency | Cost | When to use |
|------|---------|------|-------------|
| Cache lookup | <5ms | ~$0 | Repeated/similar queries |
| BM25 only | <10ms | ~$0 | Simple entity/keyword lookup |
| BM25 + Vector | <50ms | $0.0001/query | Standard queries |
| + Reranker | <150ms | $0.001/query | Queries needing precision |
| + LLM eval loop | <2s | $0.01-0.05/query | Complex or low-confidence |
| + Web search fallback | <3s | $0.01/query | Out-of-domain queries |

Route queries to the cheapest tier that can handle them. Use a lightweight classifier (or heuristics based on query length and type) to select the tier.

**Model routing for generation**

- Simple factoid answers: small model (Haiku, GPT-4o-mini) -- fast, cheap
- Complex synthesis: large model (Opus, GPT-4o) -- slower, expensive
- Route based on query complexity classification and retrieval confidence

**Embedding cost reduction**

- Batch embedding requests (API calls charge per token, not per request)
- Cache embeddings for repeated queries
- Use dimensionality reduction (Matryoshka embeddings) for initial ANN search, full dimensions for reranking

**Infrastructure savings**

- Use quantized vector indexes (binary or scalar quantization) for 4-8x memory reduction
- Offload cold data to disk-based indexes, keep hot data in memory
- Right-size GPU instances for reranking workloads

---

# Key Takeaways

## Hybrid Search Is the Baseline

Never ship vector-only retrieval. Combine dense embeddings with BM25 sparse search and reciprocal rank fusion. Add a cross-encoder reranker for the highest-impact single improvement to any RAG pipeline.

## Close the Loop

Open-loop RAG (retrieve once, generate once) fails silently. Implement retrieval evaluation -- even a simple relevance classifier -- and route to corrective actions when confidence is low. CRAG and Self-RAG provide proven patterns.

## Match Strategy to Query

Not all queries need the same retrieval depth. Simple factoid lookups should be fast and cheap. Complex multi-hop questions warrant iterative retrieval with query decomposition. Use Adaptive RAG routing to allocate resources proportionally.

## Observe Everything

Log the full pipeline trace: query, rewrite, retrieved docs, scores, reranked order, prompt, and output. Without observability, you cannot diagnose failures, measure improvements, or detect drift. Instrument before you optimize.

## Ground Every Claim

Every statement in a RAG output should trace back to a retrieved source. Implement citation generation or post-hoc attribution. Flag unsupported claims rather than presenting them as facts. Trust is earned through verifiability.

## Optimize the Pipeline, Not Just the Model

Chunking strategy, embedding model selection, reranker choice, and caching architecture often matter more than upgrading the LLM. Profile your pipeline end-to-end before spending on a larger generator.

## The Knowledge Base Is the Primary Investment

The 2026 consensus: knowledge-base quality -- chunking, freshness, and deduplication -- matters more for RAG performance than model choice. Treat the knowledge source as the asset you invest in first; a frontier model cannot rescue a stale, noisy, or poorly-chunked corpus.

## Sources

- [RAG in 2026: Knowledge-Base Quality Over Model Choice (Techment)](https://www.techment.com/blogs/rag-in-2026/)

---

# Key Papers in Self-Correcting RAG

| Paper | Authors | Venue | Core Contribution |
|-------|---------|-------|-------------------|
| [Corrective RAG (CRAG)](https://arxiv.org/abs/2401.15884) | Yan et al. | ICLR 2024 | Retrieval evaluator triggers web search fallback when internal retrieval is poor; knowledge refinement at sentence level |
| [Self-RAG](https://arxiv.org/abs/2310.11511) | Asai et al. | ICLR 2024 | Single LM learns when to retrieve and self-evaluates via special reflection tokens; inference-time controllable |
| [Adaptive-RAG](https://arxiv.org/abs/2403.14403) | Jeong et al. | NAACL 2024 | Query complexity classifier routes to single-step, multi-step, or no-retrieval strategies |
| [RAG-Fusion](https://arxiv.org/abs/2402.03367) | Rackauckas | 2024 | Generate multiple query variants and merge results via reciprocal rank fusion for broader recall |
| [REPLUG](https://arxiv.org/abs/2301.12652) | Shi et al. | NAACL 2024 | Treat LM as black box; train retriever using LM perplexity as supervision signal |
| [Active RAG (FLARE)](https://arxiv.org/abs/2305.06983) | Jiang et al. | EMNLP 2023 | Actively decide retrieval timing using forward-looking active retrieval |
| [Self-Knowledge Guided RAG](https://arxiv.org/abs/2310.05002) | Wang et al. | EMNLP 2023 | Model assesses its own knowledge to decide when retrieval is needed vs. parametric answer |
| [IRCoT](https://arxiv.org/abs/2212.10509) | Trivedi et al. | ACL 2023 | Interleave chain-of-thought reasoning with retrieval for multi-hop question answering |

**Recommended reading order** -- Start with CRAG and Self-RAG for the core self-correction patterns. Then read Adaptive-RAG for routing. IRCoT for multi-step retrieval. RAG-Fusion for query expansion.

---

# RAG 2.0

## Retrieval that knows when it's wrong -- and fixes itself.

Next week: GraphRAG and Global Context — graph-based retrieval and corpus-wide questions.