Capstone: build, measure, explain

Slide 1 of 14Open this slide

Capstone

Putting it all together

Week 15 — Applied AI Course

Frontier models, RAG, MCP, multi-agent, evaluation, safety — applied to a real problem you scoped, built, and measured.

Sources

• Anthropic — Claude model overview
• OpenAI — o-series + GPT-5 release notes
• Google DeepMind — Gemini 2 model card

Slide 2 of 14Open this slide

The Capstone Is a Working System

Not a slide deck about a system. Not a paper review. A running, demonstrable AI-powered application that exercises at least three of the techniques covered this semester.

The deliverables

Required ingredients (pick ≥ 3)

• Retrieval (vector RAG, GraphRAG, or hybrid)
• An agentic loop with tool use
• An MCP server you built or a non-trivial MCP integration
• Multi-agent orchestration
• Human-in-the-loop pause points or memory
• Fine-tuning, distillation, or routing across model sizes
• A safety/guardrail layer with eval coverage

Scoring focus

The grading rubric weights three things heavily:

1. The problem is real. Not a toy. Not a chatbot demo. Something you'd plausibly ship
2. The evaluation is real. You have numbers — quality, latency, cost — for your problem on your corpus, not a leaderboard
3. The failure analysis is honest. Capstones that pretend nothing went wrong are scored down; capstones that confront real failure modes are scored up

What's not the capstone

• A re-implementation of a tutorial
• A list of LLM API calls strung together with no architectural thought
• An evaluation paper without a system that produced the data
• A system without an evaluation

Sources

• See docs/capstone-rubric.md in the course repo (rubric details + examples)

Slide 3 of 14Open this slide

Choose Models with Evidence

Use current provider catalogs and model cards. A model name or price from an earlier course offering is a historical example, not a deployment recommendation.

A practical selection process

Start with the simplest available model and workflow that meet the task constraints. Compare alternatives on the same data. Route to a larger model only when measured results justify the added latency and cost. Document cases where no candidate meets the required behavior.

Sources

• Current model documentation and catalogs
• Public edition update notes

Slide 4 of 14Open this slide

Five Capstone Patterns

Picking your pattern

Two questions, in order:

1. What's the most interesting thing about your problem?

If it's the data → Pattern 1 or 5. If it's the workflow → Pattern 2 or 3. If it's the cost / scale → Pattern 4.

2. What can you actually evaluate?

You need a metric. If you can't define one in advance, the pattern needs to change until you can.

• Pattern 1: accuracy + groundedness on a hand-built eval set
• Pattern 2: PR/issue resolution rate or correct-action rate
• Pattern 3: end-to-end task completion + agent-on-agent quality (review scores)
• Pattern 4: accuracy + per-token cost vs the teacher model
• Pattern 5: integration test coverage; per-tool latency

What none of them are

A chatbot. "Chatbot for my company" is not a pattern — it's an interface. The capstone is about what the chatbot does, which falls into one of the five above.

Sources

• Public capstone guide. These patterns are project suggestions; private student submissions are not part of the public materials.

Slide 5 of 14Open this slide

Six Questions to Pin Down Early

1. What model?

Don't start with Opus 4.7 because it's the best. Start with the cheapest model that plausibly works for your task. Measure. Escalate only on measured failure. See lecture 12 for the small-model strategy.

2. Stateful or stateless?

• Stateless: every request stands alone (RAG-style Q&A). Easier to scale; no memory bugs
• Stateful: the agent remembers across turns (a coding agent that knows the file you're working on). Requires a memory model — see lecture 11

If you don't need statefulness, don't pay for it.

3. One agent or many?

A common over-design: starting with multi-agent for problems that fit in one model context. Heuristics:

• One agent if: the workflow is linear and the final answer is short
• Many agents if: there are explicit role boundaries (researcher / writer / critic / executor) and decomposition reduces context-window pressure

Multi-agent adds latency, cost, and debugging surface — only when justified.

4. Where does the data come from?

• Pre-indexed corpus → RAG (vector, graph, or hybrid). See lectures 4–6
• Live API → MCP servers wrapping the API. See lectures 8–9
• User upload at runtime → file handling, vision pipelines
• Memory from past sessions → memory store. See lecture 11

Most real capstones combine 2–3 of these. Be explicit about which.

5. Where does the action go?

If your agent only reads, the deployment story is easy. If it writes (sends a Slack message, creates a Jira ticket, modifies a database, opens a PR) the story gets harder fast. Plan auth, idempotency, and rollback before you write the tool, not after.

6. Who confirms destructive actions?

For any tool that has real-world side effects, decide:

• Always allow (read-only, no risk)
• Always ask (every destructive call goes to the user)
• Confirm above threshold (e.g., emails to fewer than 10 people auto-send; more, ask)

MCP's permission policy is the right place to enforce this. See lecture 8.

Anti-pattern: deciding all of this at code-time

Write a one-page architecture sketch before writing code. Diagram it. Then build.

Sources

• Anthropic — Tool use permission policies
• See lectures 3 (dev stack), 8 (MCP), 11 (memory) for component-level guidance

Slide 6 of 14Open this slide

You Need Numbers, Not Vibes

The single biggest predictor of a high-scoring capstone: a measurable eval set built early, tracked through every commit.

A working eval looks like this

@dataclass
class EvalCase:
    id: str
    input: dict          # query, file, whatever your agent takes
    gold_output: str     # human-written reference
    rubric: list[str]    # bullets the answer must address

async def run_eval(agent, cases, judge_model="claude-sonnet-4-6"):
    results = []
    for c in cases:
        out = await agent.run(c.input)
        score = await llm_judge(
            model=judge_model,
            input=c.input,
            gold=c.gold_output,
            rubric=c.rubric,
            actual=out,
        )
        results.append({"id": c.id, "score": score.value,
                         "reasoning": score.reasoning, "out": out})
    return results

What to measure

What not to measure

• Lines of code. Negative correlation with quality at this stage
• Benchmark numbers transferred from papers. Only relevant for the benchmark's task
• Tokens used per turn. Easy to optimize; rarely matters; can come at the expense of quality

Pre-commit eval gates

Some capstones add an eval gate to their git pre-commit hook: refuse to merge if the eval drops more than 2 points. This is the highest-leverage discipline in any of the past well-scored capstones — even when the gate is informal.

A practical pitfall

Building the eval set is more boring than building the agent. So people skip it. It will determine your grade. Allocate at least 25% of your time to eval design and execution.

Sources

• Anthropic — evaluating LLM applications
• Ragas — open-source eval framework
• Week 7 (Evaluation & LLM-as-a-Judge) — the long version of this lecture

Slide 7 of 14Open this slide

Estimate Before You Build

Capstones rarely fail because the model is too weak. They sometimes fail because the architecture chosen costs $5 per request and the demo budget allows for $0.05.

The cost-of-a-task calculation

For any agentic system, total cost per task ≈

Σ (input_tokens × $in + output_tokens × $out) × n_calls

For a typical Pattern-2 coding agent:

• 30k input tokens per turn (system + tools + context + history)
• 1k output tokens per turn
• 5 turns to finish a task
• Sonnet 4.6 ($3 in / $15 out)

⇒ 5 × (30000 × 3e-6 + 1000 × 15e-6) = 5 × (0.09 + 0.015) = $0.53 / task

Where the cost concentrates

• Long agentic loops — N turns of high context blows the budget. Compaction (lecture 11) or smaller models for sub-tasks help
• High output volumes — $15/M output × 10k tokens = $0.15 per call. Use structured outputs to bound
• Re-embedding on retrieval — embedding the query is cheap; re-embedding the entire corpus per change is expensive
• GraphRAG indexing — one-time but real. See lecture 6

Two cost-reducing patterns

1. Tiered models

90% of traffic served by the cheapest tier; only the hard 10% escalates.

2. Aggressive prompt caching

Anthropic's API caches identical system prompts at ~10% of normal cost on cache reads. A 5k-token system prompt re-used across 1000 requests saves ~$13.50. See prompt caching docs.

What to include in your writeup

A small table:

Sources

• Anthropic — pricing
• Anthropic — prompt caching

Slide 8 of 14Open this slide

Before You Demo

A short, opinionated checklist. Hitting all of these doesn't make your system safe; failing any of them makes it definitely unsafe.

Inputs

• User input is never concatenated into the system prompt
• Tool results are treated as untrusted text for prompt-injection purposes
• Long-running tool outputs are bounded (max bytes returned)
• Personally identifiable information is redacted from logs by default

Actions

• Every destructive tool has a confirmation step (lecture 8, permission policies)
• Outbound writes (email, Slack, PR creation) are dry-run capable
• No tool runs SQL without a read-only enforcement layer
• Idempotency keys on any external writes that could be retried

Models

• You explicitly check stop_reason == "refusal" and handle it (lecture 14)
• Adversarial inputs from your eval set don't produce destructive outputs
• If you use multiple models, you don't pass model output from one model into another's tool call without sanitization

Operations

• You have an emergency stop (kill switch / circuit breaker) for the agent loop
• You can roll back deployed prompt changes without redeploying code
• You log every tool call with timestamp, args summary, result summary
• You can identify which model version produced which output, ex post

Two things to skip if you don't have time

• Building your own jailbreak eval. It's a research project. Use Anthropic's safety eval prompts or HarmBench as a baseline
• Custom content moderation. Llama-Guard or a hosted classifier (Anthropic, OpenAI moderation API) is good enough for capstone scope

What to not skip

• Permission policies on tools with side effects
• Logging
• A clearly-documented emergency stop

Sources

• Anthropic — safety best practices
• Llama-Guard 3
• HarmBench
• Week 14 (Safety & Red-Teaming) — the long version

Slide 9 of 14Open this slide

Showing the System Off

15 minutes is tight. The goal is to show what the system does well and to confront honestly where it fails.

A working structure

Show three scenarios, not one

• The happy path — proves it works at all
• A non-trivial case — shows it goes beyond the demo script
• A failure case you didn't fix — shows you understand the limits

Skipping the third is the most common scoring miss. Showing it builds credibility.

Live vs recorded demo

Live: more impressive, riskier. Practice it three times before class. Recorded: safer, less impressive. Cut judiciously — never edit out failures, only edit out latency

A recorded demo with live narration is one option. Label recordings clearly and preserve failure cases.

What kills demos

• Live coding during the presentation (don't)
• Reading the system prompt out loud (the audience doesn't care; show the behavior)
• Spending 8 minutes on architecture and 5 on the demo (inverse the proportion)
• Live API keys on screen
• Showing credentials, private records, or sensitive command history in a shared terminal

A small detail that matters

Have a backup. If the live demo fails, you should have a 30-second recorded clip of the same scenario working that you can switch to without breaking flow. Explain when a recorded backup was captured and which scenario it demonstrates.

Sources

• Public capstone guide. Private student recordings are not included in the public materials.

Slide 10 of 14Open this slide

The Section That Distinguishes Good From Great

The capstone writeup has a section called "Failure Analysis." It is graded heavily. Past capstones that scored A vs B+ usually diverged on this section, not on the system itself.

What it isn't

• A list of features you didn't ship
• An apology for not solving everything
• A discussion of LLM limitations in general

What it is

• A categorization of the failure modes you observed
• Concrete examples of each, with input → output → why
• An honest assessment of which are fixable with engineering and which require fundamentally different choices
• An assessment of how frequent each failure mode is, in your real eval

A worked example

"23 of 50 evaluation cases produced unsatisfactory outputs. Categorized:

• 12 retrieval misses (the right document existed in the corpus but ranked outside top-10). Fixable with a better reranker — tried voyage-rerank-2 on a subset and recovered 8 of 12.
• 6 reasoning errors (correct context retrieved; model concluded wrong). 4 of these were one-step multi-hop; might benefit from explicit chain-of-thought scaffolding.
• 3 tool errors (model called update_record with stale ID after a long context). Could fix with stricter tool schema or explicit state-refresh tool.
• 2 refusals on edge-case inputs that the safety guardrail flagged as risky but were benign in context."

That paragraph is worth ~5% of the capstone grade.

How to gather the data

1. Run your eval set
2. For every case scored < 3 / 5, classify by hand into one of your categories
3. Pick 1–2 representative examples per category for the writeup
4. Don't paraphrase — quote the actual inputs and outputs

Two things to avoid

• "It works most of the time." That's not analysis. Measure "most".
• Pretending all failures are model failures. Most aren't — most are pipeline failures (chunking, retrieval ranking, prompt design). Be specific

Sources

• See docs/capstone-rubric.md (failure-analysis dimension)

Slide 11 of 14Open this slide

Common Capstone Mistakes

Scope mistakes

• Choosing too broad a problem. "Build an AI doctor" can't be evaluated. "Diagnose three categories of patient intake forms from sample data" can
• Choosing a problem with no available data. If you can't get a corpus or labeled examples by week 12, you can't evaluate. Switch
• Choosing a problem already solved by a single API call. "Summarize a PDF" — Claude does this in one call. Your system has to add value beyond that

Engineering mistakes

• Building before architecting. A 30-minute architecture sketch saves weeks
• Optimizing the wrong dimension. People micro-tune prompts for marginal quality gains while their retrieval misses 40% of the cases
• Reinventing what an MCP server already does. Assess existing integrations before implementing another one; test their behavior and access controls

Eval mistakes

• Skipping the eval set. Without evaluation data, claims about quality cannot be assessed
• Evaluating on the same data the system was tuned on. Hold-out split is non-negotiable
• LLM-judge contamination. If your judge model is the same as your generator, scores inflate. Use a different family

Demo mistakes

• No live demo. "Here are some screenshots of when it worked" — not enough
• Long architecture lecture, short demo. Audience wants to see the thing do something
• No failure case shown. Include a failure scenario so reviewers can assess the limits

Writeup mistakes

• No numbers. Every claim ("worked well", "fast", "cheap") needs a number
• No failure analysis. See previous slide
• Architecture diagram with everything labeled "LLM." Specify which model, which framework, which version
• Demo screenshots without the failure column. Always show both

One meta-mistake

Treating the capstone as a presentation about an AI system rather than an AI system that you happen to present. The system is the deliverable. The presentation is the proof that the system exists and works.

Sources

• docs/capstone-rubric.md, especially the "deliverable weight" table

Slide 12 of 14Open this slide

How You'll Be Scored

What an A looks like

• Runs end-to-end on first try from a clean clone
• Has 50+ evaluation cases with measured quality + cost + latency
• Failure analysis names 3–5 specific failure modes with frequencies
• Demo includes a scenario that doesn't work, explained
• Writeup reads like an engineering design doc, not a school assignment

What a B looks like

• Runs but with manual setup steps
• Has 10–20 eval cases or vague qualitative eval
• Failure analysis is general (one paragraph, no specific cases)
• Demo is happy-path only
• Writeup describes what was built but not the alternatives considered

What a C looks like

• Doesn't run end-to-end without instructor intervention
• No eval set
• No failure analysis or it amounts to "the LLM sometimes gets things wrong"
• Demo is screenshots only
• Writeup is a feature list

Soft expectations not in the rubric

• Citations. Don't pretend you invented techniques you didn't. Cite the papers and tools you used
• Reproducibility. Pin model versions and library versions. Include seeds where applicable
• Cost transparency. Per-task cost belongs in your writeup, even if your numbers are small

What to prioritize

• The system being smaller than you originally planned. Shipping a tight, well-evaluated subset beats a sprawling, broken whole
• The system using existing components. Building on microsoft/graphrag, LangGraph, an MCP server you didn't write — all fine, as long as you explain why and measure your additions
• The model choice being a smaller model. Evaluate whether it meets the task requirements before choosing a larger one

Sources

• Public capstone guide uses the table in this slide as its source. See docs/capstone-rubric.md for the publication arrangement.

Slide 13 of 14Open this slide

Plan the Capstone

These milestones describe a suggested self-study sequence. Choose your own dates. The public edition has no submission service, staffed office hours, or instructor grading.

Artifacts to prepare

• Code: a Git repository with dependencies and setup instructions another person can follow
• Writeup: 4–6 pages with architecture, alternatives, measured results, and limits
• Demonstration: up to 15 minutes showing normal behavior, a difficult case, and a failure
• Evaluation: CSV or JSON with per-case results and a description of the scoring method

Resources actually available

The public repository provides lecture Markdown and the slide viewer. Classroom starter projects, sample databases, private recordings, and student submissions are not included. For a new implementation, use official documentation and disposable data sources; see the resources page.

Review questions

• Can someone reproduce the result from your instructions?
• Are the evaluation data separate from the examples used during development?
• Which failures have you observed, and which paths remain untested?
• Can a reviewer understand your model choice and system boundaries?

Sources

• Public capstone guide

Slide 14 of 14Open this slide

Keep Building and Measuring

What to take away

A working mental model of how to build, evaluate, and operate AI systems. Tools and model lineups change; the discipline of defining a task, measuring behavior, and explaining failures stays useful.

Three habits to keep

1. Measure before optimizing. Establish a task metric and a baseline.
2. Make boundaries explicit. Define who can authorize an action and what happens when a dependency fails.
3. Report the evidence. Record quality, latency, cost, and the limits of what you tested.

Continue your study

• Course schedule
• Capstone guide
• Edition and version notes
• Contact the instructor

The public edition is self-study. Current university offering details and assessment arrangements belong to the relevant university course.