# Week 4: Knowledge representation
Applied AI Course — Sawan Ruparel
Public self-study edition. See https://applied-ai.sawanruparel.com/updates/ for version notes.
# Knowledge Representation
## From Embeddings to Context Management
**Applied AI Course -- Spring 2026**
Week 4
---
# The Knowledge Problem
LLMs are impressive generalists -- but they don't know **your** data.
## What LLMs Know
- General world knowledge up to training cutoff
- Common programming patterns, APIs, and frameworks
- Published scientific literature, textbooks, Wikipedia
- General reasoning and language understanding
- Public information available at scale on the internet
## What You Actually Need
- Your company's internal documentation and policies
- Private codebases, architecture decisions, runbooks
- Customer data, support tickets, CRM records
- Real-time data: stock prices, sensor feeds, logs
- Domain-specific knowledge not well-represented in training data
- Information created **after** the model's training cutoff
> The gap between general knowledge and actionable, domain-specific knowledge is the central problem of applied AI.
---
# Three Approaches to the Knowledge Gap
## Fine-Tuning
- Bake knowledge directly into model weights
- Best for: style, format, domain vocabulary
- **Not ideal for factual recall** -- models still hallucinate fine-tuned facts
- Expensive: requires curated datasets, GPU hours, and ongoing maintenance
- Latency: days to weeks per update cycle
- Example: fine-tuning on legal language for contract analysis
## Retrieval-Augmented Generation (RAG)
- Retrieve relevant documents at inference time, inject into prompt
- Best for: factual Q&A, knowledge that changes frequently
- No model retraining required -- update the index, not the model
- Cost: embedding + storage + retrieval infrastructure
- Latency: adds 100-500ms per query for retrieval step
- The **dominant pattern** in production AI systems today
## Long Context
- Stuff everything into a massive context window (128K-2M tokens)
- Best for: small-to-medium corpora, document analysis tasks
- Simple architecture: no retrieval pipeline to maintain
- Cost: proportional to tokens processed per request
- Constraint: context windows have limits; 1M tokens is ~3,000 pages
- Emerging: Gemini 1.5 Pro (2M), Claude (200K), GPT-4o (128K)
## Sources
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)](https://arxiv.org/abs/2005.11401)
---
# Mental Model: From Unstructured to Computable
The core task: turn **unstructured knowledge** into **computable representations** that an LLM can use at inference time.
```mermaid
flowchart LR
Raw["Raw knowledge
(PDFs, docs, code, APIs)"] --> Chunk["Chunked text
(semantic segments)"]
Chunk --> Vec["Dense vectors
(embeddings)"]
Vec --> Index["Indexed store
(searchable)"]
Index --> Ctx["Curated context
(right info, right order)"]
Ctx --> LLM[LLM]
```
### The Representation Pipeline
1. **Raw Knowledge** -- PDFs, docs, code, databases, APIs, web pages
2. **Chunked Text** -- meaningful segments with preserved context
3. **Dense Vectors** -- numerical representations capturing semantic meaning
4. **Indexed Store** -- searchable structure optimized for similarity retrieval
5. **Curated Context** -- the right information, in the right order, at the right time
### Key Insight
The quality of your AI application is bounded by the quality of your knowledge representation. Garbage in, hallucinations out.
### What We'll Cover Today
- **Embeddings**: how we turn text into vectors (the representation)
- **Vector Databases**: how we store and search those vectors (the infrastructure)
- **Context Management**: how we select and organize retrieved information (the intelligence)
---
# What Are Embeddings?
An **embedding** is a dense vector representation of text in a continuous high-dimensional space, where semantic similarity corresponds to geometric proximity.
### The Intuition
- The word "king" and "queen" are close in embedding space
- "How do I reset my password?" and "I forgot my login credentials" map to nearby vectors
- Unrelated concepts are far apart
### Properties of Good Embeddings
- **Semantic similarity**: similar meanings produce similar vectors
- **Compositionality**: sentence embeddings capture meaning beyond individual words
- **Task transferability**: embeddings trained on one task generalize to others
- **Dimensionality**: typically 256-3072 dimensions (tradeoff: expressiveness vs cost)
### From Words to Sentences
- **Word2Vec / GloVe** (2013-2014): word-level, static embeddings
- **ELMo** (2018): contextualized word embeddings
- **Sentence-BERT** (2019): sentence-level embeddings via siamese networks
- **Modern embedding models** (2023+): instruction-tuned, multi-task, matryoshka representations
### Why This Matters for RAG
Embeddings are the **bridge** between natural language queries and your stored knowledge. The quality of your embeddings directly determines retrieval quality.
## Sources
- [Efficient Estimation of Word Representations in Vector Space — Word2Vec (Mikolov et al., 2013)](https://arxiv.org/abs/1301.3781)
- [GloVe: Global Vectors for Word Representation (Pennington et al., 2014)](https://nlp.stanford.edu/projects/glove/)
- [Deep Contextualized Word Representations — ELMo (Peters et al., 2018)](https://arxiv.org/abs/1802.05365)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (Reimers & Gurevych, 2019)](https://arxiv.org/abs/1908.10084)
---
# How Embedding Models Work
Modern embedding models are trained with **contrastive learning**: push similar pairs together, pull dissimilar pairs apart.
```mermaid
flowchart LR
subgraph "Positive pair (similar)"
A1["reset password"] --> E1[Shared encoder]
A2["forgot login"] --> E1
E1 --> V1["cos_sim = 0.95
MINIMIZE distance"]
end
subgraph "Negative pair (dissimilar)"
B1["reset password"] --> E2[Shared encoder]
B2["quarterly revenue"] --> E2
E2 --> V2["cos_sim = 0.12
MAXIMIZE distance"]
end
```
### Key Training Techniques
- **InfoNCE Loss**: for each positive pair, treat all other batch items as negatives
- **Hard Negative Mining**: select negatives that are close but semantically different (e.g., same topic, different answer)
- **Matryoshka Representation Learning (MRL)**: train embeddings that are useful at multiple dimensionalities (e.g., 256, 512, 1024, 3072)
- **Instruction tuning**: prepend task-specific instructions ("Represent this document for retrieval:") to improve task performance
### Inference
At query time, you pass text through the encoder once and get a fixed-size vector. No pairwise comparison needed -- just compute similarity against stored vectors.
## Sources
- [Representation Learning with Contrastive Predictive Coding — InfoNCE (van den Oord et al., 2018)](https://arxiv.org/abs/1807.03748)
- [Matryoshka Representation Learning (Kusupati et al., 2022)](https://arxiv.org/abs/2205.13147)
---
# Embedding Model Landscape
Choosing an embedding model is one of the highest-leverage decisions in a RAG system.
| Model | Provider | Dimensions | Max Tokens | MTEB Avg | Notes |
|-------|----------|-----------|------------|----------|-------|
| text-embedding-3-large | OpenAI | 3072 (configurable) | 8191 | 64.6 | MRL support, can truncate dims |
| text-embedding-3-small | OpenAI | 1536 (configurable) | 8191 | 62.3 | 5x cheaper than large |
| Embed v4 | Cohere | 256-1536 (Matryoshka) | 512 | 66.3 | Multimodal (interleaved text+image), int8/binary output, on Bedrock + Azure AI Foundry |
| voyage-3-large | Voyage AI | 1024 | 32000 | 67.2 | Strong on code + long docs |
| Gemini text-embedding-004 | Google | 768 | 2048 | 66.0 | Free tier available |
| NV-Embed-v2 | NVIDIA | 4096 | 32768 | 72.3 | Mistral-7B-derived; strong open-weight choice for multilingual retrieval + enterprise on-prem |
| GTE-Qwen2-7B-instruct | Alibaba | 3584 | 32768 | 70.2 | Open-source, self-hostable |
| nomic-embed-text-v1.5 | Nomic | 768 | 8192 | 62.3 | Open-source, MRL support |
| bge-m3 | BAAI | 1024 | 8192 | 63.5 | Multi-lingual, multi-granularity |
### Selection Criteria
- **Accuracy vs cost**: bigger models perform better but cost more per embedding
- **Dimensionality**: higher dims = more expressive but more storage and slower search
- **Max token length**: must cover your chunk sizes
- **Matryoshka support**: lets you trade accuracy for efficiency post-training (e.g. Cohere Embed v4 emits 256-1536 dims from one model)
- **Multimodal inputs**: models like Cohere Embed v4 embed interleaved text + images into a shared space -- one index for documents, screenshots, and diagrams
- **Self-hosted vs API**: latency, privacy, cost at scale
- **Embeddings as a service**: hosted endpoints remove the embedding pipeline entirely -- Pinecone Inference runs embedding and reranking models inside Pinecone Serverless, and MongoDB's Automated Embedding (powered by Voyage AI) generates vectors on write with one click
## Sources
- [OpenAI Embeddings — text-embedding-3 (OpenAI, 2024)](https://platform.openai.com/docs/guides/embeddings)
- [Cohere Embed Models Documentation (Cohere, 2024)](https://docs.cohere.com/docs/embeddings)
- [Cohere Embed v4 — Multimodal Embeddings Changelog (Cohere)](https://docs.cohere.com/changelog/embed-multimodal-v4)
- [Pinecone Inference — Hosted Embedding & Reranking (DataCamp)](https://www.datacamp.com/blog/the-top-5-vector-databases)
- [Voyage AI Embeddings (Voyage AI)](https://www.voyageai.com/)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (Lee et al., 2024)](https://arxiv.org/abs/2405.17428)
- [Best Embedding Models — NV-Embed-v2 for Multilingual & On-Prem (Openxcell)](https://www.openxcell.com/blog/best-embedding-models/)
- [GTE-Qwen2-7B-instruct (Alibaba NLP)](https://huggingface.co/Alibaba-NLP/gte-Qwen2-7B-instruct)
- [Nomic Embed: Training a Reproducible Long Context Text Embedder (Nussbaum et al., 2024)](https://arxiv.org/abs/2402.01613)
- [BGE M3-Embedding (Chen et al., 2024)](https://arxiv.org/abs/2402.03216)
- [MTEB: Massive Text Embedding Benchmark (Muennighoff et al., 2022)](https://arxiv.org/abs/2210.07316)
- [MTEB Leaderboard (Hugging Face)](https://huggingface.co/spaces/mteb/leaderboard)
---
# Similarity Metrics
How you measure "closeness" between vectors matters more than you think.
## The Metrics
**Cosine Similarity**
- Measures angle between vectors, ignores magnitude
- Range: [-1, 1] (1 = identical direction)
- Formula: cos(A,B) = (A . B) / (||A|| * ||B||)
- **Most common choice** -- works well when embeddings are normalized
- Invariant to document length effects on vector magnitude
**Dot Product (Inner Product)**
- Measures both angle AND magnitude
- Range: (-inf, +inf)
- Formula: A . B = sum(a_i * b_i)
- Use when magnitude carries meaning (e.g., document importance)
- Equivalent to cosine similarity when vectors are L2-normalized
**Euclidean Distance (L2)**
- Measures straight-line distance in vector space
- Range: [0, +inf) (0 = identical)
- Formula: ||A - B|| = sqrt(sum((a_i - b_i)^2))
- Sensitive to magnitude differences
- Less common in practice for text embeddings
## When to Use Which
**Use cosine similarity when:**
- Your embeddings come from different models or are unnormalized
- You care about semantic direction, not magnitude
- Default choice for most RAG systems
**Use dot product when:**
- Vectors are already L2-normalized (then it equals cosine)
- Your vector DB optimizes for dot product (many do internally)
- You want to factor in vector magnitude as a relevance signal
**Use Euclidean when:**
- Working with non-text embeddings (images, audio)
- Clustering tasks where absolute position matters
- Your vectors occupy a bounded region of space
> **Practical tip**: Most embedding APIs return normalized vectors. When normalized, cosine similarity and dot product give identical rankings. Check your model's documentation.
---
# Chunking Strategies
How you split documents into chunks is often **more important** than which embedding model you use.
## Fixed-Size Chunking
- Split every N tokens/characters with optional overlap
- Simple and predictable -- easy to reason about storage and costs
- Overlap (e.g., 10-20%) prevents information loss at boundaries
- **Problem**: splits mid-sentence, mid-paragraph, mid-thought
- Good baseline; use 256-512 tokens with 50-token overlap
- Best for: homogeneous text, quick prototyping
## Semantic Chunking
- Split where the meaning shifts, not at arbitrary boundaries
- Compute embeddings for sentences, split when cosine similarity drops below threshold
- Produces variable-length chunks aligned to topic boundaries
- More expensive: requires embedding each sentence during ingestion
- Best for: articles, reports, heterogeneous documents
## Recursive / Hierarchical
- Try large separators first (##, \n\n), fall back to smaller ones (\n, ., " ")
- Respects document structure: headers, paragraphs, sentences
- LangChain's RecursiveCharacterTextSplitter is the classic implementation
- Controllable: set max chunk size, it finds the best split points
- Best for: markdown, code, structured documents
## Document-Aware / Agentic
- Use document structure: HTML tags, PDF sections, table boundaries
- Specialized parsers for each format (e.g., table rows stay together)
- Can use an LLM to generate chunk summaries or propositions
- Highest quality but highest cost and complexity
- Best for: technical docs, legal contracts, multi-format corpora
## Sources
- [LangChain RecursiveCharacterTextSplitter (LangChain)](https://reference.langchain.com/python/langchain-text-splitters/character/RecursiveCharacterTextSplitter)
---
# Chunking Tradeoffs
There is no universally optimal chunk size. Every choice is a tradeoff.
### Chunk Size: The Central Tension
| Small chunks (128-256 tokens) | Large chunks (512-1024 tokens) |
|-------------------------------|-------------------------------|
| More precise retrieval | More context per chunk |
| Better for specific questions | Better for complex/nuanced questions |
| More chunks to store and search | Fewer chunks, lower storage costs |
| Risk: losing surrounding context | Risk: diluting the relevant signal |
| Higher recall, lower precision | Lower recall, higher precision |
### Overlap
- **Purpose**: ensure information at chunk boundaries isn't lost
- **Typical range**: 10-20% of chunk size (e.g., 50-token overlap for 512-token chunks)
- **Cost**: increases storage by the overlap percentage
- **Diminishing returns**: beyond 20% overlap, gains are minimal
### Metadata: The Secret Weapon
Attach metadata to every chunk to enable filtering and context reconstruction:
- **Source**: file name, URL, document title
- **Position**: section header, page number, chunk index within document
- **Temporal**: creation date, last modified, version
- **Structural**: document type, author, department, access level
- **Semantic**: LLM-generated summary, keywords, or propositions
### Practical Recommendations
1. Start with **recursive chunking at 512 tokens, 50-token overlap**
2. Always preserve **section headers** as metadata (or prepend to chunk)
3. **Benchmark** different strategies on your actual queries before committing
4. Consider **proposition-based chunking** for high-value use cases (decompose paragraphs into atomic facts)
---
# Measuring Embedding Quality
You cannot improve what you do not measure. Embedding quality directly determines retrieval quality.
### Intrinsic Metrics (Embedding Space)
- **Retrieval Accuracy (Recall@k)**: what fraction of relevant documents appear in the top-k results?
- **Mean Reciprocal Rank (MRR)**: how high does the first relevant result rank on average?
- **Clustering Coherence**: do documents on the same topic cluster together? (silhouette score)
- **Nearest Neighbor Overlap**: do similar queries return overlapping result sets?
### The MTEB Benchmark
The **Massive Text Embedding Benchmark** evaluates embeddings across 8 task categories:
- Classification, Clustering, Pair Classification, Reranking
- Retrieval, Semantic Textual Similarity (STS), Summarization, BitextMining
- 56+ datasets, multiple languages
- **Limitation**: MTEB scores are averages -- a model strong on STS may be weak on retrieval
### Practical Evaluation Strategy
1. **Build a golden test set**: 50-100 question-answer pairs from your actual domain
2. **Measure Recall@5 and Recall@20**: does the right chunk appear in retrieved results?
3. **Compare models head-to-head**: same chunks, same queries, different embeddings
4. **Test with real user queries**: synthetic benchmarks don't capture real distribution
5. **Track drift over time**: as your corpus grows, retrieval quality can degrade
### Common Failure Modes
- Embedding model trained on English performs poorly on multilingual content
- Short queries fail to match long, detailed chunks (length mismatch)
- Domain jargon or acronyms not represented in the embedding model's training data
- Symmetric models (STS-trained) underperform on asymmetric retrieval (query vs document)
## Sources
- [MTEB: Massive Text Embedding Benchmark (Muennighoff et al., 2022)](https://arxiv.org/abs/2210.07316)
- [MTEB Leaderboard (Hugging Face)](https://huggingface.co/spaces/mteb/leaderboard)
---
# Vector Database Landscape
A vector database is a specialized store optimized for indexing, storing, and querying high-dimensional vectors at scale.
## Pinecone
- Fully managed, serverless option available
- Scales to billions of vectors without infrastructure management
- Sparse-dense hybrid search built in
- Namespaces for multi-tenant isolation
- **Tradeoff**: vendor lock-in, no self-hosted option, cost at scale
## Weaviate
- Open-source (BSL license), cloud or self-hosted
- Built-in vectorization modules (plug in OpenAI, Cohere, etc.)
- GraphQL API; natively combines BM25 + dense vectors + metadata filtering in a **single query**
- Multi-modal: text, images, audio in same collection
- **Tradeoff**: heavier resource footprint, complex configuration
## Qdrant
- Open-source (Apache 2.0), Rust-based, high performance
- Rich filtering with payload indexes (pre-filtering)
- Quantization support: scalar, product, binary
- gRPC and REST APIs, strong Python SDK
- **Tradeoff**: smaller ecosystem than Pinecone/Weaviate
## Chroma
- Open-source, designed for developer ergonomics
- Embedded mode: runs in-process (great for prototyping)
- Simple API: add, query, delete -- minimal boilerplate
- Integrates with LangChain, LlamaIndex out of the box
- Completed a **Rust-core rewrite** delivering ~4x performance, closing much of the prior production-scale gap
- **Tradeoff**: ecosystem and distributed-scale story still maturing vs Pinecone/Qdrant
## MongoDB Atlas Vector Search
- Vectors live alongside operational documents -- one database for app data and search
- **Automated Embedding** (powered by Voyage AI): one-click vector generation on write, no separate embedding pipeline
- Hybrid search and rich metadata filtering via the aggregation pipeline
- Strong for: teams already on MongoDB wanting to avoid a second datastore
- **Tradeoff**: vector features tied to Atlas (managed) tier
## pgvector / PostgreSQL
- Vector search as a PostgreSQL extension
- Use your existing Postgres infrastructure and tooling
- HNSW and IVFFlat index types, hybrid with SQL queries
- Strong for: teams already on Postgres, moderate scale (<10M vectors)
- **Tradeoff**: not as fast as purpose-built vector DBs at large scale
## Sources
- [Pinecone Documentation (Pinecone)](https://docs.pinecone.io)
- [Weaviate Documentation (Weaviate)](https://weaviate.io/developers/weaviate)
- [Qdrant Documentation (Qdrant)](https://qdrant.tech/documentation/)
- [Chroma Documentation (Chroma)](https://docs.trychroma.com)
- [pgvector — Open-source vector similarity search for PostgreSQL](https://github.com/pgvector/pgvector)
- [Best Vector Databases in 2026: Pricing, Scale Limits & Architecture Tradeoffs (MarkTechPost, 2026)](https://www.marktechpost.com/2026/05/10/best-vector-databases-in-2026-pricing-scale-limits-and-architecture-tradeoffs-across-nine-leading-systems/)
---
# How Vector Search Works
Exact nearest-neighbor search is O(n) -- too slow at scale. Approximate Nearest Neighbor (ANN) algorithms trade a small accuracy loss for massive speed gains.
```mermaid
flowchart TB
subgraph "HNSW: greedy descent through layers"
Q[Query vector] --> L2
L2["Layer 2 (sparse): A — D"] --> L1
L1["Layer 1 (medium): A — B — D — F"] --> L0
L0["Layer 0 (dense): A-B-C-D-E-F-G-H-I-J"] --> Ans["Nearest neighbor
O(log n)"]
end
```
```mermaid
flowchart LR
Vecs[All vectors] --> KMeans["k-means partition"]
KMeans --> C1[Cluster 1]
KMeans --> C2[Cluster 2]
KMeans --> Ck[Cluster k]
Q[Query] --> Probe{Find nearest
nprobe clusters}
C1 & C2 & Ck --> Probe
Probe --> Search["Search only
within those clusters"]
Search --> Ans[Top-k results]
```
```
Product Quantization (PQ)
==========================
Original vector (1024 dims):
[0.12, 0.45, -0.33, 0.78, ..., 0.56]
Split into 8 sub-vectors of 128 dims each:
[sub1] [sub2] [sub3] [sub4] [sub5] [sub6] [sub7] [sub8]
Quantize each sub-vector to nearest centroid ID:
[ 42 ] [ 17 ] [ 8 ] [231] [105] [ 63 ] [ 99 ] [ 5 ]
Memory: 1024 floats (4KB) --> 8 bytes
~500x compression with ~5% recall loss
```
### Performance Characteristics
| Algorithm | Build Time | Query Time | Memory | Recall@10 |
|-----------|-----------|------------|--------|-----------|
| Flat (exact) | O(1) | O(n) | 1x | 100% |
| IVF | O(n) | O(n/k) | 1x | 95-99% |
| HNSW | O(n log n) | O(log n) | 1.5-2x | 97-99.5% |
| IVF-PQ | O(n) | O(n/k) | 0.01-0.1x | 90-95% |
## Sources
- [Efficient and Robust Approximate Nearest Neighbor Search Using HNSW Graphs (Malkov & Yashunin, 2018)](https://arxiv.org/abs/1603.09320)
- [FAISS — A Library for Efficient Similarity Search (Facebook Research)](https://github.com/facebookresearch/faiss)
---
# Indexing Strategies
The choice of index determines the speed/accuracy/memory tradeoff for your vector search.
### Flat Index (Exact Search)
- Brute-force comparison against every vector
- **100% recall** -- guaranteed to find the true nearest neighbors
- Practical up to ~100K vectors (depending on latency requirements)
- Use for: golden test sets, small corpora, accuracy baselines
### HNSW (Hierarchical Navigable Small World)
- **Default choice** for most production systems
- Tunable parameters: `M` (connections per node), `ef_construction` (build quality), `ef_search` (query quality)
- Higher `M` = better recall, more memory; typical range: 16-64
- Higher `ef_search` = better recall, slower queries; typical range: 100-500
- Memory overhead: ~1.5-2x the raw vector storage
- Build time: can be slow for very large datasets (100M+)
### IVF (Inverted File Index)
- Partition-based: clusters vectors, searches only relevant partitions
- Tunable: `nlist` (number of clusters), `nprobe` (clusters searched per query)
- Faster build time than HNSW, lower memory overhead
- Good for: very large datasets where memory is constrained
- Often combined with PQ for compression: IVF-PQ
### Quantization (Compression)
- **Scalar quantization**: float32 to int8 (~4x compression, minimal accuracy loss)
- **Product quantization**: ~100x compression, moderate accuracy loss
- **Binary quantization**: 32x compression, best for reranking pipelines
- Use when: storage costs dominate or you need to fit billions of vectors in memory
### Decision Framework
1. **< 100K vectors**: flat index is fine
2. **100K - 10M vectors**: HNSW with default parameters
3. **10M - 1B vectors**: HNSW + scalar quantization, or IVF-PQ
4. **> 1B vectors**: IVF-PQ with distributed sharding
## Sources
- [Efficient and Robust Approximate Nearest Neighbor Search Using HNSW Graphs (Malkov & Yashunin, 2018)](https://arxiv.org/abs/1603.09320)
- [FAISS — A Library for Efficient Similarity Search (Facebook Research)](https://github.com/facebookresearch/faiss)
---
# Beyond Dense Vectors: Sparse & Hybrid
Dense embeddings capture meaning but miss exact terms -- product IDs, error codes, proper nouns. Two representation strategies bridge the gap:
- **Keyword / sparse (BM25):** exact lexical matching -- perfect for codes and names, no embedding model needed. "Error E-4021" always matches "Error E-4021"; "automobile" never matches "car"
- **Learned-sparse embeddings (SPLADE, BGE-M3):** trained like embedding models but emit *sparse* vectors that carry both lexical and semantic signal -- stored and searched like a keyword index. Increasingly first-class in Pinecone, Qdrant, and Weaviate
Combining dense + sparse signals ("hybrid search") consistently beats either alone -- dense for meaning, sparse for exact matches.
> This slide is about how knowledge is **represented**. The retrieval-side mechanics -- fusion (Reciprocal Rank Fusion), tuning, and the production hybrid pipeline -- are covered in **Week 5 (RAG 2.0)**.
## Sources
- [SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking (Formal et al., 2021)](https://arxiv.org/abs/2107.05720)
- [BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity (Chen et al., 2024)](https://arxiv.org/abs/2402.03216)
- [Okapi BM25 (Wikipedia)](https://en.wikipedia.org/wiki/Okapi_BM25)
---
# Metadata Filtering
Real-world retrieval is almost never "find the most similar vectors." It's "find the most similar vectors **within these constraints**."
### Why Metadata Filtering Matters
- Multi-tenant apps: only search documents belonging to the current user
- Temporal relevance: only retrieve documents from the last 90 days
- Access control: respect document permissions at retrieval time
- Domain scoping: search only within "engineering docs" or "HR policies"
### Pre-Filtering vs Post-Filtering
| Approach | How It Works | Pros | Cons |
|----------|-------------|------|------|
| **Pre-filter** | Apply metadata filter first, then vector search on subset | Guarantees all results match filter; accurate top-k | Can be slow if filter is very selective (tiny subset) |
| **Post-filter** | Vector search first, then filter results | Fast vector search; simple implementation | May return fewer than k results; wastes compute |
- **Qdrant, Weaviate**: optimized pre-filtering with payload indexes
- **Pinecone**: pre-filtering with metadata indexes
- **pgvector**: combine WHERE clauses with vector similarity (natural SQL pre-filtering)
### Best Practices for Metadata
1. **Index high-cardinality fields** you'll filter on frequently (user_id, department, doc_type)
2. **Use structured types**: dates as timestamps (not strings), categories as enums
3. **Avoid over-filtering**: very narrow filters reduce the candidate pool and hurt recall
4. **Store retrieval-relevant metadata only** in the vector DB; keep full documents elsewhere
5. **Plan for access control early**: retrofitting tenant isolation is painful
### Common Metadata Schema
```
{
"source": "confluence",
"doc_type": "engineering_runbook",
"team": "platform",
"created_at": "2025-11-15T10:30:00Z",
"updated_at": "2026-01-20T14:00:00Z",
"author": "jane.doe",
"access_level": "internal",
"section_title": "Database Failover Procedure"
}
```
---
# Choosing a Vector Database
There is no single best vector DB. The right choice depends on your constraints.
```mermaid
flowchart TB
Start[Pick a vector DB] --> Scale{Scale?}
Scale -->|< 100K| Tiny[Chroma / pgvector / NumPy]
Scale -->|100K - 10M| Mid{Managed?}
Scale -->|10M - 1B| Large{Hosted?}
Scale -->|> 1B| Huge[Pinecone serverless
or sharded Qdrant/Milvus]
Mid -->|yes| Pinecone1[Pinecone / Weaviate Cloud]
Mid -->|no| Self1[pgvector / Qdrant / Weaviate]
Large -->|yes| Pinecone2[Pinecone]
Large -->|no| Self2[Qdrant / Weaviate distributed]
```
## Step 1: Define Your Scale
- **< 100K vectors**: Chroma (embedded), pgvector, or even in-memory with NumPy
- **100K - 10M vectors**: pgvector, Qdrant, Weaviate, Pinecone
- **10M - 1B vectors**: Pinecone, Qdrant (distributed), Weaviate (distributed)
- **> 1B vectors**: Pinecone serverless, custom Qdrant/Milvus clusters
## Step 2: Managed vs Self-Hosted
- **Managed** (Pinecone, Weaviate Cloud, Qdrant Cloud): zero ops overhead, pay-per-use, vendor lock-in risk
- **Self-hosted** (Qdrant, Weaviate, Milvus, pgvector): full control, data sovereignty, requires infrastructure team
- **Embedded** (Chroma, LanceDB, SQLite-VSS): runs in-process, no server needed, limited scale
## Step 3: Feature Requirements
- **Hybrid search needed?** Pinecone, Weaviate, Qdrant all support it; pgvector requires separate BM25 setup
- **Multi-tenancy?** Pinecone namespaces, Qdrant collection-per-tenant or payload filtering, Weaviate multi-tenancy
- **Rich metadata filtering?** Qdrant and Pinecone have the strongest filtering engines
- **Existing Postgres?** pgvector avoids adding a new database to your stack
## Step 4: Evaluate Total Cost
- **Storage**: raw vectors + indexes + metadata + replication
- **Compute**: queries per second, embedding dimensions, concurrent users
- **Operational**: monitoring, backups, upgrades, incident response
- **Pinecone serverless**: ~$2/month per 1M vectors (1536-dim) for storage; pay per query
- **pgvector on RDS**: included in existing Postgres costs; limited by instance size
## Step 5: Prototype and Benchmark
- Load your actual data (not synthetic benchmarks)
- Test with your actual query patterns and filters
- Measure: p50/p95 latency, recall@10, cost per 1000 queries
- Plan for 10x growth from day one
---
# Beyond Naive Retrieval: The Context Management Paradigm
"Naive RAG" -- embed query, retrieve top-k, stuff into prompt -- works for demos but fails in production.
### Where Naive RAG Breaks Down
- **Wrong chunks retrieved**: the query doesn't match the best documents semantically
- **Right chunks, wrong order**: the LLM attends more to the beginning and end of context (lost-in-the-middle effect)
- **Too much noise**: irrelevant chunks dilute the signal and increase hallucination
- **Missing context**: the retrieved chunk is a fragment; the LLM lacks surrounding information
- **Conflicting information**: multiple chunks contradict each other with no resolution
### The Shift: Retrieval to Context Management
**Retrieval** asks: "What documents are similar to the query?"
**Context management** asks: "What is the optimal set of information to place in the LLM's context window to produce the best answer?"
### The Context Management Stack
```mermaid
flowchart LR
Q[Raw query] --> Understand["1. Query
understanding"]
Understand --> Retrieve["2. Multi-stage
retrieval"]
Retrieve --> Select["3. Context
selection"]
Select --> Order["4. Context
ordering"]
Order --> Compress["5. Context
compression"]
Compress --> LLM[LLM]
```
1. **Query understanding** -- transform the raw query into better retrieval signals
2. **Multi-stage retrieval** -- cast a wide net, then narrow with reranking
3. **Context selection** -- choose which chunks to include and which to discard
4. **Context ordering** -- arrange information for optimal LLM attention
5. **Context compression** -- summarize or extract to fit within token budgets
> This is where the engineering happens. Each stage is an opportunity to improve answer quality.
## Sources
- [Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023)](https://arxiv.org/abs/2307.03172)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)](https://arxiv.org/abs/2005.11401)
---
# Context Window Strategies
The context window is your most valuable resource. Managing it well is the difference between a good and great RAG system.
### What Goes In
- **System prompt**: instructions, persona, output format constraints
- **Retrieved chunks**: the knowledge the LLM needs to answer
- **Conversation history**: prior turns for multi-turn interactions
- **Metadata**: source citations, timestamps, confidence signals
- **Examples**: few-shot demonstrations if needed
### What Stays Out
- Chunks with low relevance scores (below a threshold, not just bottom of top-k)
- Redundant information (deduplicate overlapping chunks)
- Stale data when fresher alternatives exist
- Information the user doesn't have access to (enforce permissions)
### The Lost-in-the-Middle Effect
Research (Liu et al., 2023) showed that LLMs perform best when relevant information is at the **beginning** or **end** of the context, and worst when it's buried in the middle.
**Practical implication**: place your most relevant chunks first and last, not in the middle.
### Token Budget Allocation
For a 128K-token context window, a typical allocation:
| Component | Tokens | Notes |
|-----------|--------|-------|
| System prompt | 500-2,000 | Instructions and formatting |
| Retrieved context | 2,000-8,000 | 4-16 chunks at ~500 tokens each |
| Conversation history | 1,000-4,000 | Last 3-5 turns |
| Output reservation | 1,000-4,000 | Space for the model's response |
| **Buffer** | remaining | Headroom for edge cases |
### Ordering Strategies
1. **Relevance-first**: highest similarity score at the top
2. **Chronological**: oldest to newest (for temporal reasoning)
3. **Sandwich**: most relevant at start and end, less relevant in middle
4. **Grouped by source**: cluster chunks from the same document together
## Sources
- [Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023)](https://arxiv.org/abs/2307.03172)
---
# Reranking: From Recall to Precision
Embedding-based retrieval casts a wide net (high recall); a reranker narrows it to the most relevant results (high precision).
- **Bi-encoders** (embedding models): encode query and document *independently* -- fast, but no cross-attention between them
- **Cross-encoders** (rerankers): process query AND document *together* through full attention -- 100-1000x slower, but capture fine-grained relevance
- The standard pattern is two-stage: retrieve 50-100 candidates with the bi-encoder, then rerank to the top 5-10 with a cross-encoder
> Reranking is fundamentally a **retrieval-pipeline** technique. The reranker model landscape (Cohere, bge, ColBERT late-interaction) and where reranking sits in a production RAG pipeline are covered in **Week 5 (RAG 2.0)**.
## Sources
- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (Santhanam et al., 2022)](https://arxiv.org/abs/2112.01488)
---
# Query Transformation
The user's query is often a poor retrieval signal. Transform it before searching.
## HyDE (Hypothetical Document Embeddings)
- Ask the LLM to **generate a hypothetical answer**, then embed THAT instead of the query
- Intuition: a hypothetical answer looks more like the stored documents than a short question does
- Example: "What is HNSW?" becomes a generated paragraph explaining HNSW, which better matches documentation
- **Cost**: one LLM call per query (~200-500ms)
- Works well for technical/factual queries; less useful for conversational ones
## Multi-Query Retrieval
- Generate **3-5 query variations** from different angles using an LLM
- Retrieve results for each variation, then deduplicate and merge
- Example: "How do I handle rate limits?" generates: "rate limiting best practices", "API throttling strategies", "429 error handling"
- **Benefit**: covers more of the embedding space, higher recall
- **Cost**: multiplies retrieval calls (but retrieval is cheap)
## Step-Back Prompting
- Generate a **more abstract version** of the query before retrieving
- "Why does my Python script fail with OOM on large CSV files?" steps back to "memory management for large file processing in Python"
- Retrieves broader conceptual context, not just narrow matches
- Combine with the original query for both specific and general results
## Query Decomposition
- Break **complex questions** into sub-questions, retrieve for each
- "Compare HNSW and IVF for a 100M vector dataset with real-time requirements" decomposes into: "HNSW performance at 100M scale", "IVF performance at 100M scale", "real-time vector search requirements"
- Especially useful for multi-hop reasoning and comparison questions
- **Pattern**: decompose, retrieve per sub-query, synthesize answer from all results
## Sources
- [Precise Zero-Shot Dense Retrieval without Relevance Labels — HyDE (Gao et al., 2022)](https://arxiv.org/abs/2212.10496)
- [Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models (Zheng et al., 2023)](https://arxiv.org/abs/2310.06117)
---
# Parent-Child Retrieval: Retrieve Small, Return Large
**Core insight**: small chunks are better for matching, but large chunks are better for answering.
```mermaid
flowchart LR
subgraph "Indexed structure"
P1["Parent 1
(~1024 tok)"]
P2["Parent 2
(~1024 tok)"]
C11["Child 1.1
(128 tok)"] --> P1
C12["Child 1.2"] --> P1
C13["Child 1.3"] --> P1
C21["Child 2.1"] --> P2
C22["Child 2.2"] --> P2
C23["Child 2.3"] --> P2
end
Q["Query:
How does failover work?"] --> Search[Embed + search
against CHILD chunks]
C11 & C12 & C13 & C21 & C22 & C23 --> Search
Search --> Match["Match: Child 2.2
(score 0.89)"]
Match --> Return["Return PARENT 2
to the LLM"]
P2 -.-> Return
```
### How It Works
1. **Index small chunks** (128-256 tokens) for precise semantic matching
2. **Store parent references**: each child chunk links to its parent (the full section or document)
3. **At query time**: search against child chunks, but return the parent chunk to the LLM
4. The LLM gets the full context surrounding the matched passage
### Variations
- **Sentence window**: retrieve the matching sentence, expand to +/- N surrounding sentences
- **Section-level parents**: children are paragraphs, parents are full sections
- **Document summary + chunks**: store a summary of the whole document as metadata; include it alongside retrieved chunks
### Benefits
- Precise retrieval (small chunks match better)
- Rich context for generation (large chunks provide better answers)
- Reduces "out of context" hallucinations from truncated chunks
---
# Contextual Retrieval
Anthropic's approach (September 2024): use an LLM to add context to each chunk **before** embedding it.
### The Problem
A chunk like "The company was founded in 1983" is meaningless without knowing **which** company. Standard chunking strips away the surrounding context that makes a chunk interpretable.
### The Approach
For each chunk, prepend a short contextual description generated by an LLM:
**Before** (raw chunk):
> "Revenue increased 15% year-over-year, driven primarily by expansion in the APAC region."
**After** (contextualized chunk):
> "This chunk is from Acme Corp's Q3 2025 earnings report, specifically the revenue breakdown section. Revenue increased 15% year-over-year, driven primarily by expansion in the APAC region."
### Implementation
1. For each chunk, send the **full document + the chunk** to an LLM
2. Prompt: "Give a short context (1-2 sentences) situating this chunk within the overall document"
3. Prepend the generated context to the chunk before embedding
4. Use prompt caching to avoid re-processing the full document for each chunk (major cost savings)
### Results (Anthropic's Benchmarks)
| Method | Recall Improvement vs Baseline |
|--------|-------------------------------|
| Contextual embeddings only | 35% reduction in retrieval failures |
| Contextual embeddings + BM25 hybrid | 49% reduction in retrieval failures |
| Contextual embeddings + BM25 + reranking | 67% reduction in retrieval failures |
### Practical Considerations
- **Cost**: one LLM call per chunk at ingestion time (use prompt caching and a fast model)
- **Latency**: ingestion is slower; retrieval is unchanged
- **When to use**: high-value corpora where retrieval quality is critical
- **When to skip**: rapidly changing data where re-ingestion cost is prohibitive
- Works with any embedding model -- it improves the text being embedded, not the model itself
## Sources
- [Introducing Contextual Retrieval (Anthropic, 2024)](https://www.anthropic.com/news/contextual-retrieval)
---
# Long Context: Does RAG Still Matter?
With 1M-token context windows now production-available (GPT-5.5, Gemini 3.1 Pro), do we still need retrieval pipelines? The frontier has moved from "can we fit it?" to "should we?"
## The Case for RAG
- **Scale**: 1M tokens is ~3,000 pages; many corpora are much larger
- **Cost**: processing 1M tokens per query is expensive ($3-15+ per call depending on model)
- **Latency**: large context = slower time-to-first-token (10-30s for very long contexts)
- **Accuracy**: needle-in-a-haystack tests show degradation in very long contexts
- **Freshness**: easy to update the index without reprocessing everything
- **Access control**: can filter by permissions at retrieval time
- **Auditability**: you know exactly which chunks informed the answer
## The Case for Long Context
- **Simplicity**: no chunking, no embeddings, no vector DB, no retrieval pipeline
- **Full document understanding**: the model sees the entire document, not fragments
- **Cross-reference**: can reason across sections that RAG might not retrieve together
- **Single large docs**: for one big document, long context is now viable end-to-end
- **Small corpora**: economics favor retrieval below ~50 pages of repeatedly-queried content; above that, per-query token cost dominates
- **Complex reasoning**: tasks requiring holistic document understanding (e.g., legal analysis)
- **Rapid prototyping**: go from zero to working system in minutes, not days
### The Pragmatic View: They're Complementary
| Scenario | Best Approach |
|----------|--------------|
| Single large doc, infrequent queries | Long context |
| < ~50 pages, repeatedly queried | RAG (economics favor retrieval; amortize embedding cost) |
| Large corpus, high query volume | RAG (or RAG + long context for retrieved chunks) |
| Real-time data feeds | RAG with streaming ingestion |
| Complex document analysis | Long context (or RAG to select, then long context to analyze) |
| Multi-document synthesis | RAG to retrieve, long context to reason |
> The future is likely **RAG as a pre-filter for long context**: retrieve the 50 most relevant pages, load them all into a large context window, let the model reason over the full set.
## Sources
- [Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023)](https://arxiv.org/abs/2307.03172)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)](https://arxiv.org/abs/2005.11401)
- [AI Model Comparisons: Long-Context Windows & Retrieval Economics (GuruSup)](https://gurusup.com/blog/ai-comparisons)
---
# Designing the Ingestion Pipeline
The ingestion pipeline transforms raw documents into a searchable knowledge base. Every production RAG system needs one.
```mermaid
flowchart LR
subgraph "Raw sources"
PDF[PDFs]
Docs[Docs]
HTML[HTML]
API[APIs]
DB[Databases]
end
PDF & Docs & HTML & API & DB --> Load["LOAD
parse, extract meta,
detect format"]
Load --> Chunk["CHUNK
split, add metadata,
(optional contextualize)"]
Chunk --> Embed["EMBED
batch via embedding API,
handle failures"]
Embed --> Store["STORE
upsert to vector DB,
dedup + index build"]
```
### Stage Details
| Stage | Tools | Key Decisions |
|-------|-------|--------------|
| **Load** | Unstructured.io, LlamaParse, PyMuPDF, Docling | Format detection, OCR for scanned docs, table extraction |
| **Chunk** | LangChain splitters, LlamaIndex node parsers, custom | Strategy, size, overlap, metadata preservation |
| **Embed** | OpenAI API, Cohere API, sentence-transformers | Model choice, batch size, dimensionality, error handling |
| **Store** | Pinecone, Qdrant, pgvector, Weaviate | Upsert logic, deduplication, index configuration |
### Production Concerns
- **Idempotency**: re-running the pipeline should not create duplicates (use content hashes)
- **Incremental updates**: process only new/changed documents, not the entire corpus
- **Error handling**: log failures per-document, don't let one bad PDF kill the entire run
- **Monitoring**: track chunk counts, embedding latencies, storage utilization
- **Versioning**: when you change chunking or embedding models, you need to re-index everything
## Sources
- [Unstructured — Open-source ETL for Document Processing (Unstructured.io)](https://github.com/Unstructured-IO/unstructured)
- [LlamaParse — AI Document Parsing (LlamaIndex)](https://www.llamaindex.ai/llamaparse)
- [LlamaIndex Documentation](https://developers.llamaindex.ai/)
- [LangChain Text Splitters (LangChain)](https://reference.langchain.com/python/langchain-text-splitters/character/RecursiveCharacterTextSplitter)
- [OpenAI Embeddings API (OpenAI)](https://platform.openai.com/docs/guides/embeddings)
- [Cohere Embed Models (Cohere)](https://docs.cohere.com/docs/embeddings)
- [Pinecone Documentation (Pinecone)](https://docs.pinecone.io)
- [Weaviate Documentation (Weaviate)](https://weaviate.io/developers/weaviate)
---
# Common Pitfalls in RAG Systems
These are the failure modes I see most often in production RAG deployments.
### 1. Wrong Chunk Size
- Chunks too small: retrieved text lacks context, LLM can't form coherent answers
- Chunks too large: irrelevant text dilutes the signal, wastes context window
- **Fix**: benchmark multiple sizes (256, 512, 1024) on your actual queries
### 2. Bad or Mismatched Embeddings
- Using a general-purpose model for a highly specialized domain (medical, legal, financial)
- Embedding queries and documents with different models (they must share the same vector space)
- Not testing embedding quality on your domain before committing
- **Fix**: evaluate on a golden test set before deploying; consider domain fine-tuning
### 3. Ignoring Metadata
- Storing chunks without source, date, section, or author information
- No metadata filtering: searching the entire corpus when the answer is in a specific document
- **Fix**: design your metadata schema before building the pipeline
### 4. No Reranking Stage
- Trusting bi-encoder similarity scores as the final relevance signal
- Returning top-k by embedding similarity alone
- **Fix**: add a cross-encoder reranker; even a cheap one improves precision significantly
### 5. Stuffing Too Much into Context
- Retrieving 20 chunks "just in case" when 5 would suffice
- Not deduplicating overlapping chunks
- **Fix**: use fewer, higher-quality chunks; measure answer quality vs chunk count
### 6. Not Evaluating End-to-End
- Measuring embedding similarity but not answer correctness
- No automated evaluation pipeline; relying on vibes
- **Fix**: build an eval set of question-answer pairs; measure faithfulness, relevance, and completeness
### 7. Ignoring Document Parsing Quality
- Garbage in, garbage out: PDFs with broken table extraction, missing headers, OCR errors
- **Fix**: inspect parsed output for a sample of documents; invest in parsing quality
---
# Evaluating Retrieval
You need to evaluate at **two levels**: retrieval quality (did we find the right chunks?) and generation quality (did the LLM produce a good answer?).
### Retrieval Metrics
| Metric | What It Measures | Formula | When to Use |
|--------|-----------------|---------|-------------|
| **Recall@k** | Fraction of relevant docs in top-k | relevant_in_top_k / total_relevant | Most important metric for RAG |
| **Precision@k** | Fraction of top-k that are relevant | relevant_in_top_k / k | When context window is tight |
| **MRR** | Avg reciprocal rank of first relevant result | mean(1 / rank_of_first_relevant) | When you care about the top result |
| **NDCG@k** | Quality of ranking (position-aware) | DCG@k / ideal_DCG@k | When ranking order matters |
| **Hit Rate** | Does top-k contain any relevant doc? | queries_with_hit / total_queries | Simplest metric; good starting point |
### Generation Metrics (End-to-End)
- **Faithfulness**: is the answer grounded in the retrieved context? (no hallucination)
- **Answer relevance**: does the answer address the question?
- **Context relevance**: are the retrieved chunks relevant to the question?
- **Completeness**: does the answer cover all aspects of the question?
### Evaluation Frameworks
| Framework | Approach | Notes |
|-----------|----------|-------|
| **RAGAS** | LLM-as-judge for faithfulness, relevance, context | Open-source, widely adopted |
| **DeepEval** | LLM-based metrics with statistical rigor | Supports custom metrics |
| **Braintrust** | Logging + eval platform | Production-grade, real-time evals |
| **LangSmith** | Tracing + evaluation suite | Integrated with LangChain |
| **Manual review** | Human judgment on sampled queries | Gold standard, doesn't scale |
### Building Your Evaluation Set
1. Collect **50-100 real user queries** (or realistic synthetic ones)
2. For each query, annotate the **ideal retrieved chunks** and the **ideal answer**
3. Run your pipeline, compute metrics, identify failure modes
4. Iterate on chunking, retrieval, and prompting based on failures
5. **Automate this**: run evals on every pipeline change (CI for RAG)
## Sources
- [RAGAS: Automated Evaluation of Retrieval Augmented Generation (Shahul Es et al., 2023)](https://github.com/explodinggradients/ragas)
- [DeepEval — The LLM Evaluation Framework (Confident AI)](https://github.com/confident-ai/deepeval)
- [Braintrust — AI Evaluation and Observability Platform](https://www.braintrust.dev/)
- [LangSmith Evaluation (LangChain)](https://docs.langchain.com/langsmith/evaluation)
---
# Key Takeaways
## Embeddings Are the Foundation
- Embeddings turn text into computable representations where semantic similarity equals geometric proximity
- Model choice matters: evaluate on YOUR data, not just MTEB leaderboards
- Matryoshka embeddings let you trade dimensions for cost without retraining
- Chunking strategy often has more impact than embedding model choice
## Vector Databases Are Infrastructure
- HNSW is the default algorithm; start there
- Hybrid search (vector + BM25) consistently outperforms either alone
- Metadata filtering is essential -- design your schema upfront
- Choose managed vs self-hosted based on team capacity, not just features
## Context Management Is the Differentiator
- Naive RAG (embed, retrieve, stuff) is a starting point, not a destination
- Reranking is the highest-ROI addition to a basic RAG pipeline
- Query transformation (HyDE, multi-query) improves recall significantly
- Parent-child retrieval and contextual retrieval solve the "lost context" problem
## Evaluate Relentlessly
- Build a golden test set of real queries and expected answers
- Measure retrieval metrics (Recall@k, MRR) AND generation metrics (faithfulness, relevance)
- Automate evaluation: run it on every pipeline change
- The best RAG systems are built through iterative measurement and improvement
---
# Key Papers and Resources
### Foundational Papers
| Paper | Year | Key Contribution |
|-------|------|-----------------|
| [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al.)](https://arxiv.org/abs/2005.11401) | 2020 | Introduced the RAG paradigm |
| [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (Reimers & Gurevych)](https://arxiv.org/abs/1908.10084) | 2019 | Efficient sentence embeddings via siamese networks |
| [Efficient and Robust Approximate Nearest Neighbor Search Using HNSW Graphs (Malkov & Yashunin)](https://arxiv.org/abs/1603.09320) | 2018 | The HNSW algorithm used in most vector DBs |
| [Lost in the Middle: How Language Models Use Long Contexts (Liu et al.)](https://arxiv.org/abs/2307.03172) | 2023 | Showed positional bias in LLM context attention |
| [Matryoshka Representation Learning (Kusupati et al.)](https://arxiv.org/abs/2205.13147) | 2022 | Train embeddings useful at multiple dimensionalities |
| [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction (Khattab & Zaharia)](https://arxiv.org/abs/2004.12832) | 2020 | Late interaction model bridging bi-encoder and cross-encoder |
### Technical Resources
| Resource | Description |
|----------|-------------|
| [MTEB Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) | Live embedding model benchmarks |
| [Anthropic: Introducing Contextual Retrieval (Sep 2024)](https://www.anthropic.com/news/contextual-retrieval) | Contextual chunking approach with benchmarks |
| [Pinecone Learning Center](https://docs.pinecone.io) | Practical vector DB tutorials and architecture guides |
| [LlamaIndex Documentation](https://developers.llamaindex.ai/) | Comprehensive RAG framework with advanced retrieval patterns |
| [LangChain RAG Tutorial](https://reference.langchain.com/python/langchain-text-splitters/character/RecursiveCharacterTextSplitter) | End-to-end RAG implementation guide |
| [Weaviate Blog: Hybrid Search Explained](https://weaviate.io/developers/weaviate) | Deep dive into combining vector and keyword search |
### Recommended Reading Order
1. Lewis et al. (2020) -- understand the original RAG formulation
2. Reimers & Gurevych (2019) -- how sentence embeddings work
3. Liu et al. (2023) -- why context management matters
4. Anthropic Contextual Retrieval blog post -- modern best practices
5. MTEB Leaderboard -- evaluate current model landscape
---
# Knowledge Representation
## Key Message
The quality of your AI application is bounded by the quality of your knowledge representation.
**Embed well. Retrieve smart. Manage context.**
### Next Week
Week 5: Building Production RAG Pipelines -- hands-on implementation of everything we covered today.