Evaluation and LLM-as-a-judge

Slide 1 of 31Open this slide

Evaluation & LLM-as-a-Judge

Measuring What Matters

Week 7 | Applied AI Course | Spring 2026

"If you can't measure it, you can't improve it." -- adapted from Lord Kelvin

Your RAG pipeline is only as good as your ability to prove it works.

Slide 2 of 31Open this slide

Why Evaluation Matters

The "Vibes-Based Testing" Problem

• You build a RAG pipeline, try 5 queries, it "looks good" -- ship it
• Two weeks later: hallucinations in production, angry users, broken trust
• Manual spot-checking doesn't scale beyond a handful of examples

What Goes Wrong Without Systematic Eval

• Silent regressions: a prompt tweak improves one case, breaks ten others
• Inconsistent quality: works for common queries, fails on edge cases
• No baseline: you can't tell if v2 is actually better than v1
• False confidence: cherry-picked examples create an illusion of quality

The Cost of Getting It Wrong

The Goal

Move from "it seems to work" to "we have quantitative evidence that it works, and we'll know immediately when it stops working."

Slide 3 of 31Open this slide

The Evaluation Landscape

Benchmarks

Static test suites with known answers (MMLU, HumanEval, TruthfulQA). Good for comparing base models. Limited for production app evaluation -- your domain isn't on the benchmark. MMLU is now effectively saturated (88%+ from frontier models); the community has shifted to harder live-difficulty signals like GPQA Diamond (94.3% by Gemini 3.1 Pro) and ARC-AGI-2 (85% by GPT-5.5).

Human Evaluation

Gold standard for subjective quality. Expensive ($5-15 per judgment), slow (days-weeks), hard to reproduce. Essential for calibration but not for CI/CD.

Automated Metrics

Traditional NLP metrics: BLEU, ROUGE, BERTScore. Fast and cheap. Often poorly correlated with actual quality for open-ended generation tasks.

LLM-as-a-Judge

Use a capable LLM (GPT-4, Claude) to score outputs against rubrics. 80-90% agreement with human raters. Scales to thousands of evaluations per hour at ~$0.01-0.05 each.

Sources

• MMLU: Measuring Massive Multitask Language Understanding (Hendrycks et al., 2020)
• HumanEval: Evaluating Large Language Models Trained on Code (Chen et al., 2021)
• TruthfulQA: Measuring How Models Mimic Human Falsehoods (Lin et al., 2021)
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023)
• 2026 LLM Evaluation & Benchmarking: MMLU Saturation, GPQA Diamond, ARC-AGI-2 (Zylos)

Slide 4 of 31Open this slide

The Three-Layer Evaluation Architecture

Modern evaluation stacks are layered: each layer is broader and cheaper than the next, and each one's failures feed deeper, more expensive review.

• Layer 1 -- Automated benchmarks: broad and cheap. Run on every change; catch obvious regressions across many cases.
• Layer 2 -- LLM-as-Judge: moderate depth at low cost. Score the cases that pass Layer 1 against rubrics; surface subtler quality issues.
• Layer 3 -- Human evaluation: deep but expensive. Reserve for the ambiguous, high-stakes, or judge-disagreement cases that Layer 2 flags.

Each layer feeds its failures to the next, so expensive human attention is spent only where cheaper layers cannot decide.

From Model-Level to System-Level Evaluation

2026 frameworks have shifted from scoring individual responses to scoring full agent pipelines end-to-end:

• End-to-end task success: did the system actually complete the user's goal?
• Tool-call correctness: were the right tools called with the right arguments?
• Latency and cost: operational quality is now a first-class eval metric, not an afterthought

A model that produces great individual answers can still fail as a system -- wrong tool, too slow, too expensive. Evaluate the pipeline, not just the response.

Sources

• LLM Evaluation Frameworks: 2025 vs 2026 -- The Three-Layer Architecture (MLAI Digital)
• The Best LLM Evaluation Tools of 2026 -- System-Level Evaluation (Online Inference)

Slide 5 of 31Open this slide

What to Measure

The Five Pillars of LLM Evaluation

1. Accuracy / Correctness

   • Does the output contain factually correct information?
   • Can it be verified against a ground truth or reference?

2. Faithfulness / Groundedness

   • Is the output grounded in the provided context?
   • Does it avoid adding information not present in the source?

3. Relevance

   • Does the output actually answer the question asked?
   • Is the retrieved context relevant to the query?

4. Helpfulness / Completeness

   • Is the answer thorough enough to be useful?
   • Does it address all parts of a multi-part question?

5. Safety / Harmlessness

   • Does the output avoid toxic, biased, or harmful content?
   • Does it refuse appropriately when it should?

Choosing Your Metrics

• RAG systems: prioritize faithfulness and context quality
• Chat assistants: prioritize helpfulness and safety
• Code generation: prioritize correctness and executability
• Summarization: prioritize faithfulness and completeness

Not every metric matters equally for every application. Define what "good" means for YOUR use case first.

Slide 6 of 31Open this slide

RAG-Specific Metrics

RAG evaluation requires measuring TWO distinct stages independently.

Retrieval Quality

Did we find the right documents?

• Context Precision: What fraction of retrieved chunks are actually relevant?
  • Precision = relevant_retrieved / total_retrieved
• Context Recall: Did we retrieve all the relevant chunks?
  • Recall = relevant_retrieved / total_relevant
• Mean Reciprocal Rank (MRR): Is the most relevant chunk ranked first?
  • MRR = 1/rank_of_first_relevant
• NDCG@k: Are relevant chunks ranked higher than irrelevant ones?

Generation Quality

Given the retrieved context, is the answer good?

• Faithfulness: Is every claim in the answer supported by the context?
  • Faithfulness = supported_claims / total_claims
• Answer Relevancy: Does the answer address the original question?
• Answer Correctness: Does the answer match a known ground truth?
  • Combines factual similarity + semantic similarity
• Hallucination Rate: What fraction of claims are fabricated?

Key insight: A bad answer can come from bad retrieval OR bad generation. You must measure both to know where to fix.

Slide 7 of 31Open this slide

The RAGAS Framework

Retrieval Augmented Generation Assessment

RAGAS (Es et al., 2023) provides a standardized set of metrics specifically designed for RAG pipeline evaluation.

Core Metrics

Why RAGAS?

• Reference-free metrics: Faithfulness and answer relevancy don't need ground truth labels -- you can run them on production traffic
• LLM-based scoring: Uses an LLM to decompose answers into claims and verify each one
• Component-level diagnosis: Pinpoints whether failures come from retrieval or generation
• Framework integrations: Works with LangChain, LlamaIndex, Haystack out of the box

The RAGAS Score

A single composite score combining the core metrics:

RAGAS Score = Harmonic Mean(Faithfulness, Answer Relevancy,
                            Context Precision, Context Recall)

Typical production targets: RAGAS Score > 0.7 for general QA, > 0.85 for high-stakes domains.

Sources

• RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)
• RAGAS Documentation

Slide 8 of 31Open this slide

Faithfulness

Is the Answer Grounded in the Retrieved Context?

Faithfulness is the single most important metric for RAG systems. A high faithfulness score means the LLM is not hallucinating beyond the provided context.

How RAGAS Computes Faithfulness

Step 1: Claim Decomposition The LLM breaks the generated answer into individual atomic claims.

Answer: "Paris is the capital of France. It has a population
         of 2.1 million and was founded in the 3rd century BC."

Claims: [
  "Paris is the capital of France",
  "Paris has a population of 2.1 million",
  "Paris was founded in the 3rd century BC"
]

Step 2: Claim Verification Each claim is checked against the retrieved context for support.

Context mentions: capital status (YES), population (YES),
                  founding date (NO -- not in context)

Supported: 2 / 3

Step 3: Score Calculation

Faithfulness = number_of_supported_claims / total_claims
Faithfulness = 2 / 3 = 0.67

Interpreting Faithfulness Scores

• 1.0: Every claim in the answer is supported by context (ideal)
• 0.7-0.9: Mostly grounded, some minor additions
• < 0.5: Significant hallucination -- the model is inventing information

Common Causes of Low Faithfulness

• Context is retrieved but the model ignores it and relies on parametric knowledge
• The model over-elaborates with details not in the retrieved chunks
• Incorrect summarization or paraphrasing that changes meaning

Sources

• RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)
• RAGAS Documentation — Faithfulness

Slide 9 of 31Open this slide

Context Precision & Recall

Did we retrieve the right information?

Context Precision

Are the retrieved chunks relevant to the question?

Measures signal-to-noise ratio in your retrieval results.

Precision@k =
  (relevant chunks in top-k) / k

Weighted precision (RAGAS version):

CP = (1/k) * SUM[ (precision@i * rel_i) ]
     for i = 1..k where rel_i in {0,1}

Penalizes relevant docs ranked lower.

Low precision means: retrieving too much irrelevant context, which can confuse the generator and increase latency/cost.

Context Recall

Did we retrieve all the information needed to answer?

Requires ground truth answers for comparison.

Recall = (GT statements attributable
          to context) / (total GT statements)

RAGAS decomposes the ground truth answer into statements, then checks if each can be attributed to the retrieved context.

Low recall means: missing critical information. The generator can't produce a correct answer if the right chunks were never retrieved.

The precision-recall tradeoff: Retrieving more chunks (higher k) tends to improve recall but hurt precision. Typical sweet spot: k=3-5 for most RAG applications.

Sources

• RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)
• RAGAS Documentation

Slide 10 of 31Open this slide

Answer Relevancy & Correctness

Is the Answer Actually Useful?

Answer Relevancy

Measures whether the generated answer addresses the original question (not whether it's factually correct).

How it works (RAGAS):

1. Given the answer, use an LLM to generate N hypothetical questions that the answer could be responding to
2. Compute cosine similarity between each generated question and the original question
3. Average the similarities

Answer Relevancy = (1/N) * SUM[ cosine_sim(q_original, q_generated_i) ]

Example:

• Question: "What is the return policy?"
• Answer: "Our company was founded in 1995 in Seattle..."
• Generated questions would be about company history, NOT return policies
• Low similarity with original question = low relevancy score

Answer Correctness

Combines factual overlap with semantic similarity against a ground truth reference.

Answer Correctness = w1 * F1_factual + w2 * semantic_similarity

Where F1_factual is computed by:

• Decomposing both answer and ground truth into claims
• Computing TP (matching claims), FP (extra claims), FN (missing claims)
• F1 = 2 * TP / (2*TP + FP + FN)

Default weights: w1 = 0.75, w2 = 0.25 (factual accuracy weighted higher).

When to Use Each

• Answer Relevancy: Always -- doesn't require ground truth, catches off-topic answers
• Answer Correctness: When you have labeled test sets with expected answers

Sources

• RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)
• RAGAS Documentation

Slide 11 of 31Open this slide

End-to-End RAG Evaluation Pipeline

Key point: You need four columns of data -- question, answer, contexts, and (optionally) ground_truth -- to run the full RAGAS evaluation suite.

Sources

• RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)
• RAGAS Documentation

Slide 12 of 31Open this slide

Building Evaluation Datasets

Your evaluation is only as good as your dataset. Here's how to build one.

Step 1: Manual Curation (Start Here)

• Domain experts write 50-100 question-answer pairs covering key use cases
• Include easy questions, hard questions, edge cases, and adversarial inputs
• Label which source documents contain the answer
• This is your "golden set" -- treat it like test fixtures in software

Tip: Categorize questions by type (factual lookup, multi-hop reasoning, comparison, summarization) to diagnose failures.

Step 2: Synthetic Generation (Scale Up)

• Use an LLM to generate question-answer pairs from your document corpus
• RAGAS TestsetGenerator can create diverse question types automatically
• Filter and validate synthetic examples with human review (keep ~60-70%)
• Target: 200-500 examples for meaningful statistical power

from ragas.testset import TestsetGenerator
generator = TestsetGenerator.from_langchain(generator_llm, critic_llm)
testset = generator.generate_with_langchain_docs(documents, test_size=200)

Step 3: Production Logging (Stay Current)

• Log real user queries, retrieved contexts, and generated answers
• Sample and annotate production traffic weekly (thumbs up/down + corrections)
• Add interesting failures to your golden set
• Track distribution shift: are users asking questions you didn't anticipate?

Step 4: Continuous Maintenance

• Review and update eval datasets quarterly
• Remove stale questions when source documents change
• Add new categories as your application evolves
• Version your eval datasets alongside your code

Target composition: 30% curated golden set, 40% validated synthetic, 30% annotated production samples.

Sources

• RAGAS Documentation — Test Set Generation

Slide 13 of 31Open this slide

LLM-as-a-Judge

Using One LLM to Evaluate Another

The Core Idea

Instead of writing heuristic scoring functions or hiring human annotators, use a capable LLM to read an output and score it against defined criteria.

Why This Works

• Large LLMs have internalized human preferences from RLHF training
• They can follow detailed rubrics consistently across thousands of examples
• They provide natural language explanations for their scores (auditable)
• They can evaluate subjective qualities (tone, helpfulness) that rules can't capture

The Key Insight

RAGAS already does this under the hood. Every RAGAS metric uses an LLM as a judge to decompose claims, verify support, and assess relevance. LLM-as-a-Judge is the engine powering modern automated evaluation.

When to Use LLM-as-a-Judge

• Evaluating open-ended generation where there's no single "right answer"
• Assessing subjective qualities: helpfulness, tone, clarity, safety
• Scaling evaluation beyond what human reviewers can handle
• Running evaluation in CI/CD pipelines on every code change

Sources

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023)
• RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)

Slide 14 of 31Open this slide

Why LLM-as-a-Judge Works

High Correlation with Human Judgment at Lower Cost

Benchmark Evidence

Research consistently shows strong agreement between LLM judges and human evaluators:

Cost and Speed Comparison

The Practical Tradeoff

• LLM judges are not perfect -- they reach 80-90% agreement with humans at 500-5000x lower cost
• But human annotators only agree with EACH OTHER ~80-85% of the time (inter-annotator agreement)
• LLM judges are consistent -- same input always gets similar scores (no bad days, no fatigue)
• Use human eval to calibrate and validate your LLM judge, then scale with the LLM

2026 Best Practice

• Prefer an ensemble of judge models over a single judge -- it averages out individual model biases (see Multi-Judge Panels)
• Calibrate judges against human ratings on a held-out set before trusting them in production; report the agreement number, don't assume it

Sources

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023)
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (Liu et al., 2023)
• RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)
• Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models (Kim et al., 2023)
• Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (Dubois et al., 2024)
• LLM Evaluation in 2026: Judge Ensembles & Human Calibration (Meta-Intelligence)

Slide 15 of 31Open this slide

Judge Patterns

Three fundamental approaches to LLM-based evaluation.

Pairwise Comparison

Given a question, compare Answer A vs Answer B. The judge picks a winner or declares a tie.

Pros: Easier for LLMs than absolute scoring. Powers Chatbot Arena, AlpacaEval, MT-Bench.

Prompt: "Which response better answers the question? Choose A, B, or Tie. Explain your reasoning."

Best for: Model comparison, A/B testing prompt variants.

Pointwise Scoring

Rate a single answer on a numeric scale (1-5 or 1-10) against defined criteria.

Pros: Simpler to implement, gives absolute scores you can threshold on. Used by G-Eval, RAGAS internally.

Prompt: "Rate this response from 1-5 on [criterion]. 1 = completely fails, 5 = excellent. Provide your score and reasoning."

Best for: CI/CD gating, regression testing, monitoring dashboards.

Reference-Based Grading

Compare the answer against a known correct reference answer.

Pros: Most objective -- the judge checks factual overlap, not just vibes. Used for answer correctness in RAGAS.

Prompt: "Given the reference answer, score the candidate answer on factual accuracy and completeness (1-5)."

Best for: QA systems with known answers, exam-style evaluation.

Sources

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023)
• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (Chiang et al., 2024)
• Length-Controlled AlpacaEval (Dubois et al., 2024)
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (Liu et al., 2023)
• RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)

Slide 16 of 31Open this slide

Designing Judge Prompts

Rubrics, Criteria, and Scoring Scales

Anatomy of a Good Judge Prompt

A well-designed judge prompt has five components:

1. Role: "You are an expert evaluator assessing the quality of AI-generated responses."
2. Criteria: Exactly what dimension(s) to evaluate (one at a time is best)
3. Rubric: Concrete descriptions for each score level
4. Input format: Where to find the question, answer, context, reference
5. Output format: Structured scoring with mandatory reasoning before the score

Example: Faithfulness Rubric

Score 5: Every claim is directly supported by the provided context.
Score 4: Nearly all claims supported; one minor unsupported detail.
Score 3: Most claims supported, but 1-2 significant unsupported claims.
Score 2: Multiple unsupported claims that materially affect accuracy.
Score 1: The response contains fabricated information not in the context.

Best Practices

• One criterion per prompt: Evaluating helpfulness AND safety in one call degrades both
• Chain-of-thought first: Require the judge to explain reasoning BEFORE giving a score (reduces random scoring)
• Anchor with examples: Include 1-2 scored examples in the prompt to calibrate
• Use odd scales: 1-5 works better than 1-4 (avoids forced middle) or 1-10 (too granular, inconsistent)
• Structured output: Use JSON output format for reliable parsing

{
  "reasoning": "The answer correctly states X and Y from the context,
                but the claim about Z is not supported...",
  "score": 3
}

Slide 17 of 31Open this slide

Known Biases in LLM Judges

Understanding the Failure Modes

LLM judges are powerful but systematically biased. You must know these biases to build reliable evaluation.

Position Bias

• In pairwise comparisons, LLMs prefer the response presented first (or sometimes last)
• GPT-4 shows ~10-15% position bias in MT-Bench experiments
• The same response can win or lose depending on whether it's shown as "A" or "B"

Verbosity Bias

• LLMs tend to prefer longer, more detailed responses regardless of accuracy
• A verbose but partially wrong answer may score higher than a concise correct one
• Measured at 5-10% bias toward longer outputs in controlled studies

Self-Preference Bias (Self-Enhancement)

• LLMs rate their own outputs ~10% higher than outputs from other models
• GPT-4 judging GPT-4 outputs vs Claude outputs shows measurable self-preference
• This makes same-model evaluation unreliable for model comparison

Formatting Bias

• Responses with markdown formatting, bullet points, and headers score higher
• Well-structured wrong answers can outscore poorly-formatted correct answers

Authority/Confidence Bias

• Responses that sound confident and authoritative score higher
• Hedging language ("I think", "it might be") is penalized even when appropriate

Sensitivity to Prompt Wording

• Small changes in the judge prompt can shift scores by 0.5-1.0 points on a 5-point scale
• The order of criteria in the rubric affects which criteria get weighted more heavily

Sources

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023)

Slide 18 of 31Open this slide

Mitigating Judge Bias

Practical Techniques for More Reliable LLM Evaluation

1. Swap Positions (for Pairwise)

Run every comparison twice with A and B swapped. Only count a win if the same response wins in both orderings. Ties when results conflict.

result_1 = judge(question, answer_a=X, answer_b=Y)  # X first
result_2 = judge(question, answer_a=Y, answer_b=X)  # Y first
final = X_wins if both agree else Tie

Reduces position bias from ~15% to ~3%.

2. Use Detailed Rubrics

Vague criteria ("rate quality 1-5") amplify all biases. Specific rubrics with concrete examples for each score level reduce variance by 20-30%.

3. Calibrate with Human Labels

• Score 100+ examples with both human raters and the LLM judge
• Compute Cohen's kappa or Spearman correlation
• If agreement < 0.6, revise your rubric or switch judge models
• Re-calibrate quarterly as models update

4. Use a Different Model as Judge

Avoid self-preference bias by using a different model family as judge. Common pattern: use GPT-4o to judge Claude outputs, or vice versa.

5. Multi-Metric Decomposition

Instead of one holistic "quality" score, evaluate multiple specific dimensions separately. Aggregating 4-5 narrow scores is more reliable than one broad score.

6. Control for Length

Add explicit instructions: "Do not prefer longer responses. A concise correct answer should score higher than a verbose partially-correct one."

Validation Checklist

• Position bias test: < 5% swing when swapping A/B
• Length correlation: Spearman between score and length < 0.3
• Human agreement: Cohen's kappa > 0.6
• Test-retest reliability: same inputs produce same scores (>90%)

Sources

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023)

Slide 19 of 31Open this slide

Multi-Judge Panels

Aggregate Multiple LLM Judges for Robustness

The Idea

Just as academic papers use multiple reviewers, use multiple LLM judges and aggregate their scores. This reduces individual model biases and increases reliability.

Panel Configurations

Same model, different prompts:

scores = [
    judge(rubric_v1, answer),   # Rubric focused on accuracy
    judge(rubric_v2, answer),   # Rubric focused on completeness
    judge(rubric_v3, answer),   # Rubric focused on clarity
]
final_score = median(scores)

Different models, same prompt:

scores = [
    gpt4o_judge(rubric, answer),      # GPT-4o
    claude_judge(rubric, answer),      # Claude 3.5 Sonnet
    gemini_judge(rubric, answer),      # Gemini Pro
]
final_score = majority_vote(scores)  # or median

Aggregation Strategies

Practical Considerations

• Cost: 3 judges = 3x cost. Budget ~$0.03-0.15 per evaluation
• Latency: Run judges in parallel to avoid 3x latency
• Diminishing returns: 3 judges captures most of the benefit; 5+ judges rarely worth the cost
• Disagreement signal: When judges disagree strongly, flag for human review

The PoLL Protocol (Verga et al., 2024)

Panel of LLM Evaluators showed that a panel of smaller, diverse models outperforms a single large model judge. Three GPT-3.5-level models in a panel achieved higher human agreement than a single GPT-4 judge, at lower cost.

Sources

• Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models (Verga et al., 2024)

Slide 20 of 31Open this slide

RAGAS Deep Dive

Setup, Metrics, and Integration

Installation and Basic Usage

pip install ragas langchain-openai

from ragas import evaluate
from ragas.metrics import (
    faithfulness, answer_relevancy,
    context_precision, context_recall,
    answer_correctness
)
from datasets import Dataset

# Prepare your evaluation data
eval_data = {
    "question": ["What is the refund policy?", ...],
    "answer": ["You can return items within 30 days...", ...],
    "contexts": [["Policy doc: Returns accepted within 30 days..."], ...],
    "ground_truth": ["Items can be returned within 30 days for a full refund.", ...]
}
dataset = Dataset.from_dict(eval_data)

# Run evaluation
results = evaluate(
    dataset,
    metrics=[faithfulness, answer_relevancy,
             context_precision, context_recall],
)
print(results)  # {'faithfulness': 0.82, 'answer_relevancy': 0.91, ...}

LangChain Integration

from ragas.integrations.langchain import EvaluatorChain

faithfulness_chain = EvaluatorChain(metric=faithfulness)
result = faithfulness_chain.invoke({
    "query": question, "result": answer, "source_documents": docs
})

Configuration Options

• LLM choice: Default is GPT-4o. Switch to Claude or open-source models via LangchainLLMWrapper
• Embeddings: Default is OpenAI. Swap for local embeddings to reduce cost
• Batch processing: Use evaluate() with raise_exceptions=False for robust batch runs
• Async support: evaluate() runs LLM calls concurrently for 5-10x speedup

Cost Estimation

Budget: ~$12-24 to evaluate 200 samples across all metrics.

Sources

• RAGAS: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023)
• RAGAS Documentation

Slide 21 of 31Open this slide

DeepEval

Unit Testing for LLM Applications

Philosophy

DeepEval treats LLM evaluation like software testing: write test cases, define assertions, run them in CI/CD, get pass/fail results.

Core Concepts

pip install deepeval

from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    FaithfulnessMetric,
    AnswerRelevancyMetric,
    HallucinationMetric,
    GEval
)

# Define a test case
test_case = LLMTestCase(
    input="What is our vacation policy?",
    actual_output="Employees get 20 days of PTO per year.",
    retrieval_context=["HR Policy: Full-time employees receive 20 days PTO."],
    expected_output="Full-time employees receive 20 days of paid time off."
)

# Define metrics with thresholds
faithfulness = FaithfulnessMetric(threshold=0.7)
relevancy = AnswerRelevancyMetric(threshold=0.7)

# Assert -- fails the test if below threshold
assert_test(test_case, [faithfulness, relevancy])

Integration with pytest

# test_rag.py -- runs with: deepeval test run test_rag.py
import pytest
from deepeval import assert_test

@pytest.mark.parametrize("test_case", eval_dataset)
def test_faithfulness(test_case):
    assert_test(test_case, [FaithfulnessMetric(threshold=0.7)])

G-Eval: Custom Criteria

correctness = GEval(
    name="Correctness",
    criteria="Determine if the output is factually correct.",
    evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT,
                       LLMTestCaseParams.EXPECTED_OUTPUT],
    threshold=0.5
)

Key Features

• Confident AI dashboard: Free hosted platform to track eval results over time
• 50+ research-backed metrics: deterministic and LLM-as-Judge -- faithfulness, hallucination, bias, toxicity, summarization, and more
• CI integration: pass/fail gates that run on every code change, like a unit-test suite
• Synthesizer: Generate test datasets from documents (similar to RAGAS TestsetGenerator)
• Conversational metrics: Evaluate multi-turn conversations, not just single QA pairs

Together with RAGAS and TruLens, DeepEval forms the standard open-source evaluation triad recommended for production teams.

Sources

• DeepEval GitHub Repository
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (Liu et al., 2023)
• LLM Evaluation Frameworks Compared — DeepEval, RAGAS, TruLens (Atlan)

Slide 22 of 31Open this slide

Phoenix / Arize

Observability-Driven Evaluation and Tracing

The Observability Approach

Phoenix (by Arize AI, open-source) combines tracing and evaluation -- you instrument your RAG pipeline to capture every step, then evaluate the captured traces.

Tracing: See What Your Pipeline Actually Does

pip install arize-phoenix opentelemetry-instrumentation

import phoenix as px
from phoenix.otel import register
from openinference.instrumentation.langchain import LangChainInstrumentor

# Launch Phoenix UI
px.launch_app()

# Auto-instrument LangChain
tracer_provider = register(project_name="my-rag-app")
LangChainInstrumentor().instrument(tracer_provider=tracer_provider)

# Now every LangChain call is traced automatically
result = rag_chain.invoke("What is the return policy?")
# Phoenix UI shows: query -> retriever -> reranker -> LLM -> response

Evaluation on Traces

from phoenix.evals import llm_classify, RAG_RELEVANCY_PROMPT

# Pull traces from Phoenix
trace_df = px.Client().get_spans_dataframe()

# Run LLM-as-judge evaluation on retrieved documents
results = llm_classify(
    dataframe=trace_df,
    template=RAG_RELEVANCY_PROMPT,
    model=OpenAIModel(model="gpt-4o"),
    rails=["relevant", "irrelevant"]
)

Why Phoenix?

• Zero-code instrumentation: Auto-instruments LangChain, LlamaIndex, OpenAI, and more
• Visual trace explorer: See exactly which chunks were retrieved, what the prompt looked like, and what the LLM generated
• Built-in eval templates: Relevancy, hallucination, QA correctness, toxicity
• Local-first: Runs on your machine, no data leaves your environment
• Production monitoring: Track metrics over time, set up alerts for quality degradation

Sources

• Phoenix GitHub Repository (Arize AI)
• Phoenix Documentation

Slide 23 of 31Open this slide

Braintrust, Promptfoo & Other Platforms

The Eval Platform Landscape

Braintrust

• Focus: Experiment tracking and scoring for LLM applications
• Key feature: Side-by-side comparison of prompt variants with automatic scoring
• Differentiator: Strong logging and dataset management, team collaboration
• Pricing: Free tier available, paid for teams

import braintrust
experiment = braintrust.init(project="my-rag", experiment="v2-prompt")
for case in eval_dataset:
    output = rag_pipeline(case.question)
    experiment.log(input=case.question, output=output,
                   expected=case.ground_truth,
                   scores={"faithfulness": score})

Promptfoo

• Focus: CLI-first eval tool for prompt engineering and red-teaming
• Key feature: YAML-based test configuration, model comparison grids
• Differentiator: Excellent for comparing multiple models/prompts simultaneously
• Pricing: Open-source core, cloud features paid

# promptfoo config.yaml
prompts:
  - "Answer based on context: {{context}}\nQ: {{question}}"
  - "You are a helpful assistant. Context: {{context}}\nQ: {{question}}"
providers: [openai:gpt-4o, anthropic:claude-3.5-sonnet]
tests:
  - vars: { question: "What is the refund policy?", context: "..." }
    assert:
      - type: llm-rubric
        value: "Answer accurately describes the refund policy"

Other Notable Tools

How to Choose

• Just starting: RAGAS + DeepEval (free, Python-native, fast to set up)
• Prompt engineering focus: Promptfoo (great CLI, model comparison)
• Production observability: Phoenix/Arize (tracing + eval in one tool)
• Team collaboration: Braintrust or LangSmith (dashboards, sharing)

Sources

• Braintrust
• Promptfoo GitHub Repository
• LangSmith Documentation
• Weights & Biases Weave
• OpenAI Evals GitHub Repository

Slide 24 of 31Open this slide

Building Custom Evaluation Pipelines

When off-the-shelf metrics don't cover your domain requirements.

Template for custom LLM judge metric:

CUSTOM_JUDGE_PROMPT = """
You are evaluating a {domain} AI assistant.
Criterion: {criterion_name}
Definition: {criterion_definition}
Rubric:
  5 - {score_5_description}
  3 - {score_3_description}
  1 - {score_1_description}

Question: {question}
Context: {context}
Answer: {answer}

First explain your reasoning, then provide your score (1-5).
Output JSON: {"reasoning": "...", "score": N}
"""

Slide 25 of 31Open this slide

Offline vs Online Evaluation

Two complementary strategies for different stages of the development lifecycle.

Offline (Pre-Deployment)

When: Before shipping changes.

What you evaluate:

• Fixed test datasets (golden set + synthetic)
• Known question-answer pairs with ground truth
• Controlled conditions, reproducible results

How:

• RAGAS / DeepEval on curated eval sets
• Regression tests in CI/CD
• A/B comparisons of prompt variants
• Red-teaming for safety and edge cases

Strengths:

• Reproducible and comparable across versions
• Catches regressions before users see them
• Can test adversarial and edge cases safely

Weakness:

• Doesn't reflect real user behavior
• Dataset can become stale

Online (Production Monitoring)

When: After deployment, continuously.

What you evaluate:

• Real user queries and actual system responses
• Production traffic patterns and edge cases
• User satisfaction signals

How:

• Sample and score production traces with LLM judges
• Collect implicit feedback (thumbs up/down, follow-ups)
• Monitor retrieval quality metrics on live queries
• Track latency, error rates, and fallback rates

Strengths:

• Reflects actual user experience
• Catches issues offline eval missed
• Discovers new query patterns

Weakness:

• Users see failures before you catch them
• Noisier signals, harder to debug

Best practice: Gate deployments on offline eval (must pass regression tests), then monitor online eval for the first 24-48 hours after every release.

Slide 26 of 31Open this slide

Regression Testing

Ensuring Changes Don't Break Existing Behavior

The Problem

You improve your prompt for one type of question. Three other question types silently degrade. Without regression testing, you won't know until users complain.

Setting Up Regression Tests

# eval/test_regression.py
import json
from deepeval import assert_test
from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric

GOLDEN_SET = json.load(open("eval/golden_set.json"))

METRICS = [
    FaithfulnessMetric(threshold=0.7),
    AnswerRelevancyMetric(threshold=0.7),
]

def test_regression_suite():
    for case in GOLDEN_SET:
        output = rag_pipeline(case["question"])
        test_case = LLMTestCase(
            input=case["question"],
            actual_output=output["answer"],
            retrieval_context=output["contexts"],
            expected_output=case.get("ground_truth")
        )
        assert_test(test_case, METRICS)

CI/CD Integration

# .github/workflows/eval.yml
eval-regression:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - run: pip install -r requirements.txt
    - run: deepeval test run eval/test_regression.py
      env:
        OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    - run: python eval/compare_to_baseline.py

Tracking Metrics Over Time

• Store eval results as JSON artifacts alongside each commit/PR
• Compare current scores against the last release baseline
• Set absolute thresholds (faithfulness must be > 0.7) AND relative thresholds (faithfulness must not drop > 5% from baseline)
• Alert on regressions, block merges that fail eval

What to Include in Your Golden Set

• High-frequency queries: The 50 most common real user questions
• Known failure cases: Questions that broke in the past (prevent re-occurrence)
• Edge cases: Ambiguous queries, multi-hop questions, out-of-scope questions
• Safety tests: Prompt injection attempts, requests for harmful content
• Format diversity: Short questions, long questions, multi-part questions

Sources

• DeepEval GitHub Repository

Slide 27 of 31Open this slide

Human Evaluation

When You Still Need Humans in the Loop

LLM Judges Are Not Enough

LLM judges fail or are unreliable for:

• Novel domains: The judge LLM may lack domain expertise (medical, legal, scientific)
• Subjective quality: Tone, brand voice, cultural appropriateness
• Safety edge cases: Subtle harmful content that LLMs don't flag
• Calibration: You need human labels to validate that your LLM judge is actually working
• High-stakes decisions: When wrong answers have legal or financial consequences

Designing Human Evaluation

Annotation Schema

• Use the SAME rubrics you give the LLM judge (enables direct comparison)
• Binary labels (good/bad) for quick passes; Likert scales (1-5) for nuanced analysis
• Always include "unsure" option to avoid forced guessing

Sample Size Guidelines

Inter-Annotator Agreement

• Have 2-3 annotators label the same samples
• Compute Cohen's kappa (2 raters) or Fleiss' kappa (3+ raters)
• Target kappa > 0.6 (substantial agreement) before trusting labels
• If kappa < 0.4, your rubric is ambiguous -- revise it

The Hybrid Approach

1. Calibration set: 200 human-labeled examples
2. Validate LLM judge: Compare LLM scores to human labels (target agreement > 80%)
3. Scale with LLM: Use the validated LLM judge for the remaining thousands of examples
4. Ongoing audit: Human-review 5-10% of LLM-judged samples weekly to detect drift
5. Escalation: Route low-confidence LLM judgments to human reviewers

Slide 28 of 31Open this slide

Eval-Driven Development

Write Evals Before Writing Prompts

The Methodology

Eval-driven development (EDD) is test-driven development (TDD) for LLM applications. Just as TDD says "write the test first, then write the code," EDD says:

Write the eval first, then write the prompt.

The EDD Loop

1. Define what "good" looks like for your use case (criteria + rubrics)
2. Build your eval dataset (50-100 examples covering key scenarios)
3. Set up automated scoring (RAGAS / DeepEval / custom judge)
4. Write your first prompt and run evals -- establish a baseline
5. Iterate on the prompt using eval scores as your guide
6. Add failure cases to the eval set as you discover them
7. Repeat for every prompt change, model swap, or pipeline modification

Why EDD Works

• Prevents overfitting to anecdotes: You optimize for aggregate performance, not cherry-picked examples
• Makes tradeoffs visible: Improving one dimension while degrading another shows up immediately
• Enables confident refactoring: Swap models, change chunking strategies, restructure prompts -- evals tell you if it's better
• Compounds over time: Your eval set becomes a living specification of your system's requirements

Anti-Patterns to Avoid

• Eval after the fact: Building evals only after problems appear in production
• Optimizing for the eval: Overfitting prompts to pass specific test cases rather than genuinely improving quality
• Stale datasets: Never updating your eval set as your application evolves
• Single-metric obsession: Optimizing one metric (e.g., faithfulness) while ignoring others (e.g., helpfulness)
• No baseline: Making changes without recording the previous eval scores for comparison

The EDD Manifesto

• Evals are first-class artifacts, versioned and reviewed like code
• Every prompt change requires an eval run before merge
• Eval results are shared and visible to the whole team
• Failures are features: every production bug becomes a new test case

Slide 29 of 31Open this slide

Key Takeaways

Measure Both Stages

RAG evaluation requires separate metrics for retrieval quality (precision, recall) and generation quality (faithfulness, relevancy). Diagnose which component is failing before optimizing.

Use RAGAS as Your Starting Point

RAGAS provides battle-tested metrics that work out of the box. Start with faithfulness and answer relevancy (no ground truth needed), then add context metrics as you build labeled datasets.

LLM Judges Scale Evaluation

LLM-as-a-Judge achieves 80-85% human agreement at 100x lower cost. But know the biases (position, verbosity, self-preference) and mitigate them with position swapping, rubrics, and calibration.

Eval-Driven Development

Write evals before prompts. Gate deployments on regression tests. Monitor production quality continuously. Your eval dataset is a living specification that grows with your application.

Slide 30 of 31Open this slide

Key Papers & References

Essential Reading for LLM Evaluation

Benchmarks to Know

Recommended Further Reading

• Hamel Husain's blog: "Your AI Product Needs Evals" (practical eval methodology)
• Eugene Yan's "Patterns for Building LLM-based Systems & Products" (eval section)
• Anthropic's "Challenges in Evaluating AI Systems" (safety-focused evaluation)

Slide 31 of 31Open this slide

Week 7 Wrap-Up

What We Covered

RAG evaluation with RAGAS -- LLM-as-a-Judge patterns and biases -- Evaluation frameworks and tooling -- Production eval workflows

The One Thing to Remember

"You can't improve what you can't measure. In the world of LLMs, measurement itself is an LLM problem."

Next Week

Week 8: Model Context Protocol — tools, resources, prompts, transport, and authorization.

Assignment

Build a RAGAS evaluation pipeline for your Week 5/6 RAG project. Report faithfulness, answer relevancy, context precision, and context recall on a 50-question eval set. Compare at least two prompt or retrieval variants.