Knowledge representation

Slide 1 of 30Open this slide

Knowledge Representation

From Embeddings to Context Management

Applied AI Course -- Spring 2026

Week 4

Slide 2 of 30Open this slide

The Knowledge Problem

LLMs are impressive generalists -- but they don't know your data.

What LLMs Know

• General world knowledge up to training cutoff
• Common programming patterns, APIs, and frameworks
• Published scientific literature, textbooks, Wikipedia
• General reasoning and language understanding
• Public information available at scale on the internet

What You Actually Need

• Your company's internal documentation and policies
• Private codebases, architecture decisions, runbooks
• Customer data, support tickets, CRM records
• Real-time data: stock prices, sensor feeds, logs
• Domain-specific knowledge not well-represented in training data
• Information created after the model's training cutoff

The gap between general knowledge and actionable, domain-specific knowledge is the central problem of applied AI.

Slide 3 of 30Open this slide

Three Approaches to the Knowledge Gap

Fine-Tuning

• Bake knowledge directly into model weights
• Best for: style, format, domain vocabulary
• Not ideal for factual recall -- models still hallucinate fine-tuned facts
• Expensive: requires curated datasets, GPU hours, and ongoing maintenance
• Latency: days to weeks per update cycle
• Example: fine-tuning on legal language for contract analysis

Retrieval-Augmented Generation (RAG)

• Retrieve relevant documents at inference time, inject into prompt
• Best for: factual Q&A, knowledge that changes frequently
• No model retraining required -- update the index, not the model
• Cost: embedding + storage + retrieval infrastructure
• Latency: adds 100-500ms per query for retrieval step
• The dominant pattern in production AI systems today

Long Context

• Stuff everything into a massive context window (128K-2M tokens)
• Best for: small-to-medium corpora, document analysis tasks
• Simple architecture: no retrieval pipeline to maintain
• Cost: proportional to tokens processed per request
• Constraint: context windows have limits; 1M tokens is ~3,000 pages
• Emerging: Gemini 1.5 Pro (2M), Claude (200K), GPT-4o (128K)

Sources

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)

Slide 4 of 30Open this slide

Mental Model: From Unstructured to Computable

The core task: turn unstructured knowledge into computable representations that an LLM can use at inference time.

The Representation Pipeline

1. Raw Knowledge -- PDFs, docs, code, databases, APIs, web pages
2. Chunked Text -- meaningful segments with preserved context
3. Dense Vectors -- numerical representations capturing semantic meaning
4. Indexed Store -- searchable structure optimized for similarity retrieval
5. Curated Context -- the right information, in the right order, at the right time

Key Insight

The quality of your AI application is bounded by the quality of your knowledge representation. Garbage in, hallucinations out.

What We'll Cover Today

• Embeddings: how we turn text into vectors (the representation)
• Vector Databases: how we store and search those vectors (the infrastructure)
• Context Management: how we select and organize retrieved information (the intelligence)

Slide 5 of 30Open this slide

What Are Embeddings?

An embedding is a dense vector representation of text in a continuous high-dimensional space, where semantic similarity corresponds to geometric proximity.

The Intuition

• The word "king" and "queen" are close in embedding space
• "How do I reset my password?" and "I forgot my login credentials" map to nearby vectors
• Unrelated concepts are far apart

Properties of Good Embeddings

• Semantic similarity: similar meanings produce similar vectors
• Compositionality: sentence embeddings capture meaning beyond individual words
• Task transferability: embeddings trained on one task generalize to others
• Dimensionality: typically 256-3072 dimensions (tradeoff: expressiveness vs cost)

From Words to Sentences

• Word2Vec / GloVe (2013-2014): word-level, static embeddings
• ELMo (2018): contextualized word embeddings
• Sentence-BERT (2019): sentence-level embeddings via siamese networks
• Modern embedding models (2023+): instruction-tuned, multi-task, matryoshka representations

Why This Matters for RAG

Embeddings are the bridge between natural language queries and your stored knowledge. The quality of your embeddings directly determines retrieval quality.

Sources

• Efficient Estimation of Word Representations in Vector Space — Word2Vec (Mikolov et al., 2013)
• GloVe: Global Vectors for Word Representation (Pennington et al., 2014)
• Deep Contextualized Word Representations — ELMo (Peters et al., 2018)
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (Reimers & Gurevych, 2019)

Slide 6 of 30Open this slide

How Embedding Models Work

Modern embedding models are trained with contrastive learning: push similar pairs together, pull dissimilar pairs apart.

Key Training Techniques

• InfoNCE Loss: for each positive pair, treat all other batch items as negatives
• Hard Negative Mining: select negatives that are close but semantically different (e.g., same topic, different answer)
• Matryoshka Representation Learning (MRL): train embeddings that are useful at multiple dimensionalities (e.g., 256, 512, 1024, 3072)
• Instruction tuning: prepend task-specific instructions ("Represent this document for retrieval:") to improve task performance

Inference

At query time, you pass text through the encoder once and get a fixed-size vector. No pairwise comparison needed -- just compute similarity against stored vectors.

Sources

• Representation Learning with Contrastive Predictive Coding — InfoNCE (van den Oord et al., 2018)
• Matryoshka Representation Learning (Kusupati et al., 2022)

Slide 7 of 30Open this slide

Embedding Model Landscape

Choosing an embedding model is one of the highest-leverage decisions in a RAG system.

Selection Criteria

• Accuracy vs cost: bigger models perform better but cost more per embedding
• Dimensionality: higher dims = more expressive but more storage and slower search
• Max token length: must cover your chunk sizes
• Matryoshka support: lets you trade accuracy for efficiency post-training (e.g. Cohere Embed v4 emits 256-1536 dims from one model)
• Multimodal inputs: models like Cohere Embed v4 embed interleaved text + images into a shared space -- one index for documents, screenshots, and diagrams
• Self-hosted vs API: latency, privacy, cost at scale
• Embeddings as a service: hosted endpoints remove the embedding pipeline entirely -- Pinecone Inference runs embedding and reranking models inside Pinecone Serverless, and MongoDB's Automated Embedding (powered by Voyage AI) generates vectors on write with one click

Sources

• OpenAI Embeddings — text-embedding-3 (OpenAI, 2024)
• Cohere Embed Models Documentation (Cohere, 2024)
• Cohere Embed v4 — Multimodal Embeddings Changelog (Cohere)
• Pinecone Inference — Hosted Embedding & Reranking (DataCamp)
• Voyage AI Embeddings (Voyage AI)
• NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (Lee et al., 2024)
• Best Embedding Models — NV-Embed-v2 for Multilingual & On-Prem (Openxcell)
• GTE-Qwen2-7B-instruct (Alibaba NLP)
• Nomic Embed: Training a Reproducible Long Context Text Embedder (Nussbaum et al., 2024)
• BGE M3-Embedding (Chen et al., 2024)
• MTEB: Massive Text Embedding Benchmark (Muennighoff et al., 2022)
• MTEB Leaderboard (Hugging Face)

Slide 8 of 30Open this slide

Similarity Metrics

How you measure "closeness" between vectors matters more than you think.

The Metrics

Cosine Similarity

• Measures angle between vectors, ignores magnitude
• Range: [-1, 1] (1 = identical direction)
• Formula: cos(A,B) = (A . B) / (||A|| * ||B||)
• Most common choice -- works well when embeddings are normalized
• Invariant to document length effects on vector magnitude

Dot Product (Inner Product)

• Measures both angle AND magnitude
• Range: (-inf, +inf)
• Formula: A . B = sum(a_i * b_i)
• Use when magnitude carries meaning (e.g., document importance)
• Equivalent to cosine similarity when vectors are L2-normalized

Euclidean Distance (L2)

• Measures straight-line distance in vector space
• Range: [0, +inf) (0 = identical)
• Formula: ||A - B|| = sqrt(sum((a_i - b_i)^2))
• Sensitive to magnitude differences
• Less common in practice for text embeddings

When to Use Which

Use cosine similarity when:

• Your embeddings come from different models or are unnormalized
• You care about semantic direction, not magnitude
• Default choice for most RAG systems

Use dot product when:

• Vectors are already L2-normalized (then it equals cosine)
• Your vector DB optimizes for dot product (many do internally)
• You want to factor in vector magnitude as a relevance signal

Use Euclidean when:

• Working with non-text embeddings (images, audio)
• Clustering tasks where absolute position matters
• Your vectors occupy a bounded region of space

Practical tip: Most embedding APIs return normalized vectors. When normalized, cosine similarity and dot product give identical rankings. Check your model's documentation.

Slide 9 of 30Open this slide

Chunking Strategies

How you split documents into chunks is often more important than which embedding model you use.

Fixed-Size Chunking

• Split every N tokens/characters with optional overlap
• Simple and predictable -- easy to reason about storage and costs
• Overlap (e.g., 10-20%) prevents information loss at boundaries
• Problem: splits mid-sentence, mid-paragraph, mid-thought
• Good baseline; use 256-512 tokens with 50-token overlap
• Best for: homogeneous text, quick prototyping

Semantic Chunking

• Split where the meaning shifts, not at arbitrary boundaries
• Compute embeddings for sentences, split when cosine similarity drops below threshold
• Produces variable-length chunks aligned to topic boundaries
• More expensive: requires embedding each sentence during ingestion
• Best for: articles, reports, heterogeneous documents

Recursive / Hierarchical

• Try large separators first (##, \n\n), fall back to smaller ones (\n, ., " ")
• Respects document structure: headers, paragraphs, sentences
• LangChain's RecursiveCharacterTextSplitter is the classic implementation
• Controllable: set max chunk size, it finds the best split points
• Best for: markdown, code, structured documents

Document-Aware / Agentic

• Use document structure: HTML tags, PDF sections, table boundaries
• Specialized parsers for each format (e.g., table rows stay together)
• Can use an LLM to generate chunk summaries or propositions
• Highest quality but highest cost and complexity
• Best for: technical docs, legal contracts, multi-format corpora

Sources

• LangChain RecursiveCharacterTextSplitter (LangChain)

Slide 10 of 30Open this slide

Chunking Tradeoffs

There is no universally optimal chunk size. Every choice is a tradeoff.

Chunk Size: The Central Tension

Overlap

• Purpose: ensure information at chunk boundaries isn't lost
• Typical range: 10-20% of chunk size (e.g., 50-token overlap for 512-token chunks)
• Cost: increases storage by the overlap percentage
• Diminishing returns: beyond 20% overlap, gains are minimal

Metadata: The Secret Weapon

Attach metadata to every chunk to enable filtering and context reconstruction:

• Source: file name, URL, document title
• Position: section header, page number, chunk index within document
• Temporal: creation date, last modified, version
• Structural: document type, author, department, access level
• Semantic: LLM-generated summary, keywords, or propositions

Practical Recommendations

1. Start with recursive chunking at 512 tokens, 50-token overlap
2. Always preserve section headers as metadata (or prepend to chunk)
3. Benchmark different strategies on your actual queries before committing
4. Consider proposition-based chunking for high-value use cases (decompose paragraphs into atomic facts)

Slide 11 of 30Open this slide

Measuring Embedding Quality

You cannot improve what you do not measure. Embedding quality directly determines retrieval quality.

Intrinsic Metrics (Embedding Space)

• Retrieval Accuracy (Recall@k): what fraction of relevant documents appear in the top-k results?
• Mean Reciprocal Rank (MRR): how high does the first relevant result rank on average?
• Clustering Coherence: do documents on the same topic cluster together? (silhouette score)
• Nearest Neighbor Overlap: do similar queries return overlapping result sets?

The MTEB Benchmark

The Massive Text Embedding Benchmark evaluates embeddings across 8 task categories:

• Classification, Clustering, Pair Classification, Reranking
• Retrieval, Semantic Textual Similarity (STS), Summarization, BitextMining
• 56+ datasets, multiple languages
• Limitation: MTEB scores are averages -- a model strong on STS may be weak on retrieval

Practical Evaluation Strategy

1. Build a golden test set: 50-100 question-answer pairs from your actual domain
2. Measure Recall@5 and Recall@20: does the right chunk appear in retrieved results?
3. Compare models head-to-head: same chunks, same queries, different embeddings
4. Test with real user queries: synthetic benchmarks don't capture real distribution
5. Track drift over time: as your corpus grows, retrieval quality can degrade

Common Failure Modes

• Embedding model trained on English performs poorly on multilingual content
• Short queries fail to match long, detailed chunks (length mismatch)
• Domain jargon or acronyms not represented in the embedding model's training data
• Symmetric models (STS-trained) underperform on asymmetric retrieval (query vs document)

Sources

• MTEB: Massive Text Embedding Benchmark (Muennighoff et al., 2022)
• MTEB Leaderboard (Hugging Face)

Slide 12 of 30Open this slide

Vector Database Landscape

A vector database is a specialized store optimized for indexing, storing, and querying high-dimensional vectors at scale.

Pinecone

• Fully managed, serverless option available
• Scales to billions of vectors without infrastructure management
• Sparse-dense hybrid search built in
• Namespaces for multi-tenant isolation
• Tradeoff: vendor lock-in, no self-hosted option, cost at scale

Weaviate

• Open-source (BSL license), cloud or self-hosted
• Built-in vectorization modules (plug in OpenAI, Cohere, etc.)
• GraphQL API; natively combines BM25 + dense vectors + metadata filtering in a single query
• Multi-modal: text, images, audio in same collection
• Tradeoff: heavier resource footprint, complex configuration

Qdrant

• Open-source (Apache 2.0), Rust-based, high performance
• Rich filtering with payload indexes (pre-filtering)
• Quantization support: scalar, product, binary
• gRPC and REST APIs, strong Python SDK
• Tradeoff: smaller ecosystem than Pinecone/Weaviate

Chroma

• Open-source, designed for developer ergonomics
• Embedded mode: runs in-process (great for prototyping)
• Simple API: add, query, delete -- minimal boilerplate
• Integrates with LangChain, LlamaIndex out of the box
• Completed a Rust-core rewrite delivering ~4x performance, closing much of the prior production-scale gap
• Tradeoff: ecosystem and distributed-scale story still maturing vs Pinecone/Qdrant

MongoDB Atlas Vector Search

• Vectors live alongside operational documents -- one database for app data and search
• Automated Embedding (powered by Voyage AI): one-click vector generation on write, no separate embedding pipeline
• Hybrid search and rich metadata filtering via the aggregation pipeline
• Strong for: teams already on MongoDB wanting to avoid a second datastore
• Tradeoff: vector features tied to Atlas (managed) tier

pgvector / PostgreSQL

• Vector search as a PostgreSQL extension
• Use your existing Postgres infrastructure and tooling
• HNSW and IVFFlat index types, hybrid with SQL queries
• Strong for: teams already on Postgres, moderate scale (<10M vectors)
• Tradeoff: not as fast as purpose-built vector DBs at large scale

Sources

• Pinecone Documentation (Pinecone)
• Weaviate Documentation (Weaviate)
• Qdrant Documentation (Qdrant)
• Chroma Documentation (Chroma)
• pgvector — Open-source vector similarity search for PostgreSQL
• Best Vector Databases in 2026: Pricing, Scale Limits & Architecture Tradeoffs (MarkTechPost, 2026)

Slide 13 of 30Open this slide

How Vector Search Works

Exact nearest-neighbor search is O(n) -- too slow at scale. Approximate Nearest Neighbor (ANN) algorithms trade a small accuracy loss for massive speed gains.

  Product Quantization (PQ)
  ==========================

  Original vector (1024 dims):
  [0.12, 0.45, -0.33, 0.78, ..., 0.56]

  Split into 8 sub-vectors of 128 dims each:
  [sub1] [sub2] [sub3] [sub4] [sub5] [sub6] [sub7] [sub8]

  Quantize each sub-vector to nearest centroid ID:
  [ 42 ] [ 17 ] [  8 ] [231] [105] [ 63 ] [ 99 ] [  5 ]

  Memory: 1024 floats (4KB) --> 8 bytes
  ~500x compression with ~5% recall loss

Performance Characteristics

Sources

• Efficient and Robust Approximate Nearest Neighbor Search Using HNSW Graphs (Malkov & Yashunin, 2018)
• FAISS — A Library for Efficient Similarity Search (Facebook Research)

Slide 14 of 30Open this slide

Indexing Strategies

The choice of index determines the speed/accuracy/memory tradeoff for your vector search.

Flat Index (Exact Search)

• Brute-force comparison against every vector
• 100% recall -- guaranteed to find the true nearest neighbors
• Practical up to ~100K vectors (depending on latency requirements)
• Use for: golden test sets, small corpora, accuracy baselines

HNSW (Hierarchical Navigable Small World)

• Default choice for most production systems
• Tunable parameters: M (connections per node), ef_construction (build quality), ef_search (query quality)
• Higher M = better recall, more memory; typical range: 16-64
• Higher ef_search = better recall, slower queries; typical range: 100-500
• Memory overhead: ~1.5-2x the raw vector storage
• Build time: can be slow for very large datasets (100M+)

IVF (Inverted File Index)

• Partition-based: clusters vectors, searches only relevant partitions
• Tunable: nlist (number of clusters), nprobe (clusters searched per query)
• Faster build time than HNSW, lower memory overhead
• Good for: very large datasets where memory is constrained
• Often combined with PQ for compression: IVF-PQ

Quantization (Compression)

• Scalar quantization: float32 to int8 (~4x compression, minimal accuracy loss)
• Product quantization: ~100x compression, moderate accuracy loss
• Binary quantization: 32x compression, best for reranking pipelines
• Use when: storage costs dominate or you need to fit billions of vectors in memory

Decision Framework

1. < 100K vectors: flat index is fine
2. 100K - 10M vectors: HNSW with default parameters
3. 10M - 1B vectors: HNSW + scalar quantization, or IVF-PQ
4. > 1B vectors: IVF-PQ with distributed sharding

Sources

• Efficient and Robust Approximate Nearest Neighbor Search Using HNSW Graphs (Malkov & Yashunin, 2018)
• FAISS — A Library for Efficient Similarity Search (Facebook Research)

Slide 15 of 30Open this slide

Beyond Dense Vectors: Sparse & Hybrid

Dense embeddings capture meaning but miss exact terms -- product IDs, error codes, proper nouns. Two representation strategies bridge the gap:

• Keyword / sparse (BM25): exact lexical matching -- perfect for codes and names, no embedding model needed. "Error E-4021" always matches "Error E-4021"; "automobile" never matches "car"
• Learned-sparse embeddings (SPLADE, BGE-M3): trained like embedding models but emit sparse vectors that carry both lexical and semantic signal -- stored and searched like a keyword index. Increasingly first-class in Pinecone, Qdrant, and Weaviate

Combining dense + sparse signals ("hybrid search") consistently beats either alone -- dense for meaning, sparse for exact matches.

This slide is about how knowledge is represented. The retrieval-side mechanics -- fusion (Reciprocal Rank Fusion), tuning, and the production hybrid pipeline -- are covered in Week 5 (RAG 2.0).

Sources

• SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking (Formal et al., 2021)
• BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity (Chen et al., 2024)
• Okapi BM25 (Wikipedia)

Slide 16 of 30Open this slide

Metadata Filtering

Real-world retrieval is almost never "find the most similar vectors." It's "find the most similar vectors within these constraints."

Why Metadata Filtering Matters

• Multi-tenant apps: only search documents belonging to the current user
• Temporal relevance: only retrieve documents from the last 90 days
• Access control: respect document permissions at retrieval time
• Domain scoping: search only within "engineering docs" or "HR policies"

Pre-Filtering vs Post-Filtering

• Qdrant, Weaviate: optimized pre-filtering with payload indexes
• Pinecone: pre-filtering with metadata indexes
• pgvector: combine WHERE clauses with vector similarity (natural SQL pre-filtering)

Best Practices for Metadata

1. Index high-cardinality fields you'll filter on frequently (user_id, department, doc_type)
2. Use structured types: dates as timestamps (not strings), categories as enums
3. Avoid over-filtering: very narrow filters reduce the candidate pool and hurt recall
4. Store retrieval-relevant metadata only in the vector DB; keep full documents elsewhere
5. Plan for access control early: retrofitting tenant isolation is painful

Common Metadata Schema

{
  "source": "confluence",
  "doc_type": "engineering_runbook",
  "team": "platform",
  "created_at": "2025-11-15T10:30:00Z",
  "updated_at": "2026-01-20T14:00:00Z",
  "author": "jane.doe",
  "access_level": "internal",
  "section_title": "Database Failover Procedure"
}

Slide 17 of 30Open this slide

Choosing a Vector Database

There is no single best vector DB. The right choice depends on your constraints.

Step 1: Define Your Scale

• < 100K vectors: Chroma (embedded), pgvector, or even in-memory with NumPy
• 100K - 10M vectors: pgvector, Qdrant, Weaviate, Pinecone
• 10M - 1B vectors: Pinecone, Qdrant (distributed), Weaviate (distributed)
• > 1B vectors: Pinecone serverless, custom Qdrant/Milvus clusters

Step 2: Managed vs Self-Hosted

• Managed (Pinecone, Weaviate Cloud, Qdrant Cloud): zero ops overhead, pay-per-use, vendor lock-in risk
• Self-hosted (Qdrant, Weaviate, Milvus, pgvector): full control, data sovereignty, requires infrastructure team
• Embedded (Chroma, LanceDB, SQLite-VSS): runs in-process, no server needed, limited scale

Step 3: Feature Requirements

• Hybrid search needed? Pinecone, Weaviate, Qdrant all support it; pgvector requires separate BM25 setup
• Multi-tenancy? Pinecone namespaces, Qdrant collection-per-tenant or payload filtering, Weaviate multi-tenancy
• Rich metadata filtering? Qdrant and Pinecone have the strongest filtering engines
• Existing Postgres? pgvector avoids adding a new database to your stack

Step 4: Evaluate Total Cost

• Storage: raw vectors + indexes + metadata + replication
• Compute: queries per second, embedding dimensions, concurrent users
• Operational: monitoring, backups, upgrades, incident response
• Pinecone serverless: ~$2/month per 1M vectors (1536-dim) for storage; pay per query
• pgvector on RDS: included in existing Postgres costs; limited by instance size

Step 5: Prototype and Benchmark

• Load your actual data (not synthetic benchmarks)
• Test with your actual query patterns and filters
• Measure: p50/p95 latency, recall@10, cost per 1000 queries
• Plan for 10x growth from day one

Slide 18 of 30Open this slide

Beyond Naive Retrieval: The Context Management Paradigm

"Naive RAG" -- embed query, retrieve top-k, stuff into prompt -- works for demos but fails in production.

Where Naive RAG Breaks Down

• Wrong chunks retrieved: the query doesn't match the best documents semantically
• Right chunks, wrong order: the LLM attends more to the beginning and end of context (lost-in-the-middle effect)
• Too much noise: irrelevant chunks dilute the signal and increase hallucination
• Missing context: the retrieved chunk is a fragment; the LLM lacks surrounding information
• Conflicting information: multiple chunks contradict each other with no resolution

The Shift: Retrieval to Context Management

Retrieval asks: "What documents are similar to the query?"

Context management asks: "What is the optimal set of information to place in the LLM's context window to produce the best answer?"

The Context Management Stack

1. Query understanding -- transform the raw query into better retrieval signals
2. Multi-stage retrieval -- cast a wide net, then narrow with reranking
3. Context selection -- choose which chunks to include and which to discard
4. Context ordering -- arrange information for optimal LLM attention
5. Context compression -- summarize or extract to fit within token budgets

This is where the engineering happens. Each stage is an opportunity to improve answer quality.

Sources

• Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023)
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)

Slide 19 of 30Open this slide

Context Window Strategies

The context window is your most valuable resource. Managing it well is the difference between a good and great RAG system.

What Goes In

• System prompt: instructions, persona, output format constraints
• Retrieved chunks: the knowledge the LLM needs to answer
• Conversation history: prior turns for multi-turn interactions
• Metadata: source citations, timestamps, confidence signals
• Examples: few-shot demonstrations if needed

What Stays Out

• Chunks with low relevance scores (below a threshold, not just bottom of top-k)
• Redundant information (deduplicate overlapping chunks)
• Stale data when fresher alternatives exist
• Information the user doesn't have access to (enforce permissions)

The Lost-in-the-Middle Effect

Research (Liu et al., 2023) showed that LLMs perform best when relevant information is at the beginning or end of the context, and worst when it's buried in the middle.

Practical implication: place your most relevant chunks first and last, not in the middle.

Token Budget Allocation

For a 128K-token context window, a typical allocation:

Ordering Strategies

1. Relevance-first: highest similarity score at the top
2. Chronological: oldest to newest (for temporal reasoning)
3. Sandwich: most relevant at start and end, less relevant in middle
4. Grouped by source: cluster chunks from the same document together

Sources

• Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023)

Slide 20 of 30Open this slide

Reranking: From Recall to Precision

Embedding-based retrieval casts a wide net (high recall); a reranker narrows it to the most relevant results (high precision).

• Bi-encoders (embedding models): encode query and document independently -- fast, but no cross-attention between them
• Cross-encoders (rerankers): process query AND document together through full attention -- 100-1000x slower, but capture fine-grained relevance
• The standard pattern is two-stage: retrieve 50-100 candidates with the bi-encoder, then rerank to the top 5-10 with a cross-encoder

Reranking is fundamentally a retrieval-pipeline technique. The reranker model landscape (Cohere, bge, ColBERT late-interaction) and where reranking sits in a production RAG pipeline are covered in Week 5 (RAG 2.0).

Sources

• ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (Santhanam et al., 2022)

Slide 21 of 30Open this slide

Query Transformation

The user's query is often a poor retrieval signal. Transform it before searching.

HyDE (Hypothetical Document Embeddings)

• Ask the LLM to generate a hypothetical answer, then embed THAT instead of the query
• Intuition: a hypothetical answer looks more like the stored documents than a short question does
• Example: "What is HNSW?" becomes a generated paragraph explaining HNSW, which better matches documentation
• Cost: one LLM call per query (~200-500ms)
• Works well for technical/factual queries; less useful for conversational ones

Multi-Query Retrieval

• Generate 3-5 query variations from different angles using an LLM
• Retrieve results for each variation, then deduplicate and merge
• Example: "How do I handle rate limits?" generates: "rate limiting best practices", "API throttling strategies", "429 error handling"
• Benefit: covers more of the embedding space, higher recall
• Cost: multiplies retrieval calls (but retrieval is cheap)

Step-Back Prompting

• Generate a more abstract version of the query before retrieving
• "Why does my Python script fail with OOM on large CSV files?" steps back to "memory management for large file processing in Python"
• Retrieves broader conceptual context, not just narrow matches
• Combine with the original query for both specific and general results

Query Decomposition

• Break complex questions into sub-questions, retrieve for each
• "Compare HNSW and IVF for a 100M vector dataset with real-time requirements" decomposes into: "HNSW performance at 100M scale", "IVF performance at 100M scale", "real-time vector search requirements"
• Especially useful for multi-hop reasoning and comparison questions
• Pattern: decompose, retrieve per sub-query, synthesize answer from all results

Sources

• Precise Zero-Shot Dense Retrieval without Relevance Labels — HyDE (Gao et al., 2022)
• Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models (Zheng et al., 2023)

Slide 22 of 30Open this slide

Parent-Child Retrieval: Retrieve Small, Return Large

Core insight: small chunks are better for matching, but large chunks are better for answering.

How It Works

1. Index small chunks (128-256 tokens) for precise semantic matching
2. Store parent references: each child chunk links to its parent (the full section or document)
3. At query time: search against child chunks, but return the parent chunk to the LLM
4. The LLM gets the full context surrounding the matched passage

Variations

• Sentence window: retrieve the matching sentence, expand to +/- N surrounding sentences
• Section-level parents: children are paragraphs, parents are full sections
• Document summary + chunks: store a summary of the whole document as metadata; include it alongside retrieved chunks

Benefits

• Precise retrieval (small chunks match better)
• Rich context for generation (large chunks provide better answers)
• Reduces "out of context" hallucinations from truncated chunks

Slide 23 of 30Open this slide

Contextual Retrieval

Anthropic's approach (September 2024): use an LLM to add context to each chunk before embedding it.

The Problem

A chunk like "The company was founded in 1983" is meaningless without knowing which company. Standard chunking strips away the surrounding context that makes a chunk interpretable.

The Approach

For each chunk, prepend a short contextual description generated by an LLM:

Before (raw chunk):

"Revenue increased 15% year-over-year, driven primarily by expansion in the APAC region."

After (contextualized chunk):

"This chunk is from Acme Corp's Q3 2025 earnings report, specifically the revenue breakdown section. Revenue increased 15% year-over-year, driven primarily by expansion in the APAC region."

Implementation

1. For each chunk, send the full document + the chunk to an LLM
2. Prompt: "Give a short context (1-2 sentences) situating this chunk within the overall document"
3. Prepend the generated context to the chunk before embedding
4. Use prompt caching to avoid re-processing the full document for each chunk (major cost savings)

Results (Anthropic's Benchmarks)

Practical Considerations

• Cost: one LLM call per chunk at ingestion time (use prompt caching and a fast model)
• Latency: ingestion is slower; retrieval is unchanged
• When to use: high-value corpora where retrieval quality is critical
• When to skip: rapidly changing data where re-ingestion cost is prohibitive
• Works with any embedding model -- it improves the text being embedded, not the model itself

Sources

• Introducing Contextual Retrieval (Anthropic, 2024)

Slide 24 of 30Open this slide

Long Context: Does RAG Still Matter?

With 1M-token context windows now production-available (GPT-5.5, Gemini 3.1 Pro), do we still need retrieval pipelines? The frontier has moved from "can we fit it?" to "should we?"

The Case for RAG

• Scale: 1M tokens is ~3,000 pages; many corpora are much larger
• Cost: processing 1M tokens per query is expensive ($3-15+ per call depending on model)
• Latency: large context = slower time-to-first-token (10-30s for very long contexts)
• Accuracy: needle-in-a-haystack tests show degradation in very long contexts
• Freshness: easy to update the index without reprocessing everything
• Access control: can filter by permissions at retrieval time
• Auditability: you know exactly which chunks informed the answer

The Case for Long Context

• Simplicity: no chunking, no embeddings, no vector DB, no retrieval pipeline
• Full document understanding: the model sees the entire document, not fragments
• Cross-reference: can reason across sections that RAG might not retrieve together
• Single large docs: for one big document, long context is now viable end-to-end
• Small corpora: economics favor retrieval below ~50 pages of repeatedly-queried content; above that, per-query token cost dominates
• Complex reasoning: tasks requiring holistic document understanding (e.g., legal analysis)
• Rapid prototyping: go from zero to working system in minutes, not days

The Pragmatic View: They're Complementary

The future is likely RAG as a pre-filter for long context: retrieve the 50 most relevant pages, load them all into a large context window, let the model reason over the full set.

Sources

• Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023)
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
• AI Model Comparisons: Long-Context Windows & Retrieval Economics (GuruSup)

Slide 25 of 30Open this slide

Designing the Ingestion Pipeline

The ingestion pipeline transforms raw documents into a searchable knowledge base. Every production RAG system needs one.

Stage Details

Production Concerns

• Idempotency: re-running the pipeline should not create duplicates (use content hashes)
• Incremental updates: process only new/changed documents, not the entire corpus
• Error handling: log failures per-document, don't let one bad PDF kill the entire run
• Monitoring: track chunk counts, embedding latencies, storage utilization
• Versioning: when you change chunking or embedding models, you need to re-index everything

Sources

• Unstructured — Open-source ETL for Document Processing (Unstructured.io)
• LlamaParse — AI Document Parsing (LlamaIndex)
• LlamaIndex Documentation
• LangChain Text Splitters (LangChain)
• OpenAI Embeddings API (OpenAI)
• Cohere Embed Models (Cohere)
• Pinecone Documentation (Pinecone)
• Weaviate Documentation (Weaviate)

Slide 26 of 30Open this slide

Common Pitfalls in RAG Systems

These are the failure modes I see most often in production RAG deployments.

1. Wrong Chunk Size

• Chunks too small: retrieved text lacks context, LLM can't form coherent answers
• Chunks too large: irrelevant text dilutes the signal, wastes context window
• Fix: benchmark multiple sizes (256, 512, 1024) on your actual queries

2. Bad or Mismatched Embeddings

• Using a general-purpose model for a highly specialized domain (medical, legal, financial)
• Embedding queries and documents with different models (they must share the same vector space)
• Not testing embedding quality on your domain before committing
• Fix: evaluate on a golden test set before deploying; consider domain fine-tuning

3. Ignoring Metadata

• Storing chunks without source, date, section, or author information
• No metadata filtering: searching the entire corpus when the answer is in a specific document
• Fix: design your metadata schema before building the pipeline

4. No Reranking Stage

• Trusting bi-encoder similarity scores as the final relevance signal
• Returning top-k by embedding similarity alone
• Fix: add a cross-encoder reranker; even a cheap one improves precision significantly

5. Stuffing Too Much into Context

• Retrieving 20 chunks "just in case" when 5 would suffice
• Not deduplicating overlapping chunks
• Fix: use fewer, higher-quality chunks; measure answer quality vs chunk count

6. Not Evaluating End-to-End

• Measuring embedding similarity but not answer correctness
• No automated evaluation pipeline; relying on vibes
• Fix: build an eval set of question-answer pairs; measure faithfulness, relevance, and completeness

7. Ignoring Document Parsing Quality

• Garbage in, garbage out: PDFs with broken table extraction, missing headers, OCR errors
• Fix: inspect parsed output for a sample of documents; invest in parsing quality

Slide 27 of 30Open this slide

Evaluating Retrieval

You need to evaluate at two levels: retrieval quality (did we find the right chunks?) and generation quality (did the LLM produce a good answer?).

Retrieval Metrics

Generation Metrics (End-to-End)

• Faithfulness: is the answer grounded in the retrieved context? (no hallucination)
• Answer relevance: does the answer address the question?
• Context relevance: are the retrieved chunks relevant to the question?
• Completeness: does the answer cover all aspects of the question?

Evaluation Frameworks

Building Your Evaluation Set

1. Collect 50-100 real user queries (or realistic synthetic ones)
2. For each query, annotate the ideal retrieved chunks and the ideal answer
3. Run your pipeline, compute metrics, identify failure modes
4. Iterate on chunking, retrieval, and prompting based on failures
5. Automate this: run evals on every pipeline change (CI for RAG)

Sources

• RAGAS: Automated Evaluation of Retrieval Augmented Generation (Shahul Es et al., 2023)
• DeepEval — The LLM Evaluation Framework (Confident AI)
• Braintrust — AI Evaluation and Observability Platform
• LangSmith Evaluation (LangChain)

Slide 28 of 30Open this slide

Key Takeaways

Embeddings Are the Foundation

• Embeddings turn text into computable representations where semantic similarity equals geometric proximity
• Model choice matters: evaluate on YOUR data, not just MTEB leaderboards
• Matryoshka embeddings let you trade dimensions for cost without retraining
• Chunking strategy often has more impact than embedding model choice

Vector Databases Are Infrastructure

• HNSW is the default algorithm; start there
• Hybrid search (vector + BM25) consistently outperforms either alone
• Metadata filtering is essential -- design your schema upfront
• Choose managed vs self-hosted based on team capacity, not just features

Context Management Is the Differentiator

• Naive RAG (embed, retrieve, stuff) is a starting point, not a destination
• Reranking is the highest-ROI addition to a basic RAG pipeline
• Query transformation (HyDE, multi-query) improves recall significantly
• Parent-child retrieval and contextual retrieval solve the "lost context" problem

Evaluate Relentlessly

• Build a golden test set of real queries and expected answers
• Measure retrieval metrics (Recall@k, MRR) AND generation metrics (faithfulness, relevance)
• Automate evaluation: run it on every pipeline change
• The best RAG systems are built through iterative measurement and improvement

Slide 29 of 30Open this slide

Key Papers and Resources

Foundational Papers

Technical Resources

Recommended Reading Order

1. Lewis et al. (2020) -- understand the original RAG formulation
2. Reimers & Gurevych (2019) -- how sentence embeddings work
3. Liu et al. (2023) -- why context management matters
4. Anthropic Contextual Retrieval blog post -- modern best practices
5. MTEB Leaderboard -- evaluate current model landscape

Slide 30 of 30Open this slide

Knowledge Representation

Key Message

The quality of your AI application is bounded by the quality of your knowledge representation.

Embed well. Retrieve smart. Manage context.

Next Week

Week 5: Building Production RAG Pipelines -- hands-on implementation of everything we covered today.