# Week 15: Capstone: build, measure, explain Applied AI Course — Sawan Ruparel Public self-study edition. See https://applied-ai.sawanruparel.com/updates/ for version notes. # Capstone ## Putting it all together **Week 15 — Applied AI Course** Frontier models, RAG, MCP, multi-agent, evaluation, safety — applied to a real problem you scoped, built, and measured. --- Sources - [Anthropic — Claude model overview](https://www.anthropic.com/news) - [OpenAI — o-series + GPT-5 release notes](https://platform.openai.com/docs/models) - [Google DeepMind — Gemini 2 model card](https://deepmind.google/technologies/gemini/) --- # The Capstone Is a Working System Not a slide deck about a system. Not a paper review. A **running, demonstrable AI-powered application** that exercises at least three of the techniques covered this semester. ## The deliverables | Artifact | Format | Why it matters | |----------|--------|-----------------| | Working application | Git repo + README that lets a TA run it | Code that doesn't run isn't a capstone | | 15-minute demo | Live or recorded | Show the system *being used*, not described | | Architecture writeup | 4–6 pages | The thinking behind the choices, with diagrams | | Eval results | Section in the writeup | "It seems to work" is not a result | | Failure analysis | Section in the writeup | What goes wrong, what you'd fix with more time | ## Required ingredients (pick ≥ 3) - Retrieval (vector RAG, GraphRAG, or hybrid) - An agentic loop with tool use - An MCP server you built or a non-trivial MCP integration - Multi-agent orchestration - Human-in-the-loop pause points or memory - Fine-tuning, distillation, or routing across model sizes - A safety/guardrail layer with eval coverage ## Scoring focus The grading rubric weights three things heavily: 1. **The problem is real.** Not a toy. Not a chatbot demo. Something you'd plausibly ship 2. **The evaluation is real.** You have numbers — quality, latency, cost — for *your* problem on *your* corpus, not a leaderboard 3. **The failure analysis is honest.** Capstones that pretend nothing went wrong are scored down; capstones that confront real failure modes are scored up ## What's *not* the capstone - A re-implementation of a tutorial - A list of LLM API calls strung together with no architectural thought - An evaluation paper without a system that produced the data - A system without an evaluation Sources - See `docs/capstone-rubric.md` in the course repo (rubric details + examples) --- # Choose Models with Evidence Use current provider catalogs and model cards. A model name or price from an earlier course offering is a historical example, not a deployment recommendation. | Decision | Evidence to collect | |---------|---------------------| | Task quality | A held-out evaluation set and a simple baseline | | Tool use and structured output | Schema validity, action accuracy, rejected inputs, failure recovery | | Context and multimodality | Actual input sizes and modality support for the chosen model version | | Cost | Input/output usage, retries, tool calls, and cost per completed task | | Latency | End-to-end percentiles at the intended concurrency | | Hosting | Full weight memory, KV cache, runtime overhead, hardware, and operational ownership | | Data and use rights | Current service agreement, model license, data permissions, and region requirements | ```mermaid flowchart TB Task[Define the task and constraints] --> Candidates[Select available candidate models] Candidates --> Baseline[Run the same held-out evaluation] Baseline --> Quality{Meets quality and safety requirements?} Quality -->|No| Revise[Revise the model or system] Revise --> Baseline Quality -->|Yes| Compare[Compare cost, latency, and operation] Compare --> Record[Record model version and test date] ``` ## A practical selection process Start with the simplest available model and workflow that meet the task constraints. Compare alternatives on the same data. Route to a larger model only when measured results justify the added latency and cost. Document cases where no candidate meets the required behavior. Sources - [Current model documentation and catalogs](/resources/#model-catalogs-and-prices) - [Public edition update notes](/updates/) --- # Five Capstone Patterns ```mermaid flowchart TB P1["1. Domain RAG agent
e.g. medical literature Q&A"] P2["2. Code-aware agent
e.g. PR reviewer, infra agent"] P3["3. Multi-agent workflow
e.g. research team simulation"] P4["4. Fine-tuned specialist
e.g. classification at production cost"] P5["5. Domain MCP server
+ thin host"] ``` | Pattern | What it is | When to pick it | Stack typically | |---------|------------|-----------------|-----------------| | **1. Domain RAG agent** | Retrieves from a corpus you scoped, generates with citations | You have a corpus and a clear question type | Vector + reranker; optional GraphRAG; one model | | **2. Code-aware agent** | Reads a codebase, makes informed changes or recommendations | You're comfortable with the engineering | Agentic loop + MCP + workspace tools | | **3. Multi-agent workflow** | Several specialized agents coordinate (researcher + writer + critic) | The problem decomposes naturally | LangGraph or similar; explicit message bus | | **4. Fine-tuned specialist** | A small/open model fine-tuned to do one thing well, cheaply | You have labeled data and a clear metric | Llama 3.3 / Mistral + LoRA + eval harness | | **5. Domain MCP server** | A reusable capability (your data, your action) exposed via MCP | The capability is what's interesting, the host is generic | MCP server + Claude Desktop / Cursor / custom host | ## Picking your pattern Two questions, in order: ### 1. What's the most interesting thing about your problem? If it's **the data** → Pattern 1 or 5. If it's **the workflow** → Pattern 2 or 3. If it's **the cost / scale** → Pattern 4. ### 2. What can you actually evaluate? You need a metric. If you can't define one in advance, the pattern needs to change until you can. - Pattern 1: accuracy + groundedness on a hand-built eval set - Pattern 2: PR/issue resolution rate or correct-action rate - Pattern 3: end-to-end task completion + agent-on-agent quality (review scores) - Pattern 4: accuracy + per-token cost vs the teacher model - Pattern 5: integration test coverage; per-tool latency ## What none of them are **A chatbot.** "Chatbot for my company" is not a pattern — it's an interface. The capstone is about what the chatbot *does*, which falls into one of the five above. Sources - [Public capstone guide](/capstone/). These patterns are project suggestions; private student submissions are not part of the public materials. --- # Six Questions to Pin Down Early ```mermaid flowchart LR Q1[1. What model?] --> Q2[2. Stateful or stateless?] Q2 --> Q3[3. One agent or many?] Q3 --> Q4[4. Where does the data come from?] Q4 --> Q5[5. Where does the action go?] Q5 --> Q6[6. Who confirms
destructive actions?] ``` ## 1. What model? Don't start with Opus 4.7 because it's the best. Start with the cheapest model that plausibly works for your task. Measure. Escalate only on measured failure. See lecture 12 for the small-model strategy. ## 2. Stateful or stateless? - **Stateless:** every request stands alone (RAG-style Q&A). Easier to scale; no memory bugs - **Stateful:** the agent remembers across turns (a coding agent that knows the file you're working on). Requires a memory model — see lecture 11 If you don't *need* statefulness, don't pay for it. ## 3. One agent or many? A common over-design: starting with multi-agent for problems that fit in one model context. Heuristics: - One agent if: the workflow is linear and the final answer is short - Many agents if: there are explicit role boundaries (researcher / writer / critic / executor) **and** decomposition reduces context-window pressure Multi-agent adds latency, cost, and debugging surface — only when justified. ## 4. Where does the data come from? - **Pre-indexed corpus** → RAG (vector, graph, or hybrid). See lectures 4–6 - **Live API** → MCP servers wrapping the API. See lectures 8–9 - **User upload at runtime** → file handling, vision pipelines - **Memory from past sessions** → memory store. See lecture 11 Most real capstones combine 2–3 of these. Be explicit about which. ## 5. Where does the action go? If your agent only *reads*, the deployment story is easy. If it *writes* (sends a Slack message, creates a Jira ticket, modifies a database, opens a PR) the story gets harder fast. Plan auth, idempotency, and rollback before you write the tool, not after. ## 6. Who confirms destructive actions? For any tool that has real-world side effects, decide: - Always allow (read-only, no risk) - Always ask (every destructive call goes to the user) - Confirm above threshold (e.g., emails to fewer than 10 people auto-send; more, ask) MCP's permission policy is the right place to enforce this. See lecture 8. ## Anti-pattern: deciding all of this at code-time Write a one-page architecture sketch *before* writing code. Diagram it. Then build. Sources - [Anthropic — Tool use permission policies](https://docs.claude.com/en/docs/agents-and-tools/tool-use/overview) - See lectures 3 (dev stack), 8 (MCP), 11 (memory) for component-level guidance --- # You Need Numbers, Not Vibes The single biggest predictor of a high-scoring capstone: **a measurable eval set built early, tracked through every commit**. ```mermaid flowchart LR A["1. Hand-label 30–50
real queries / tasks"] --> B["2. Define one
primary metric"] B --> C["3. Define guardrail
metrics
(latency, cost, safety)"] C --> D["4. Run on every
significant change"] D --> E["5. Plot trends
in writeup"] ``` ## A working eval looks like this ```python @dataclass class EvalCase: id: str input: dict # query, file, whatever your agent takes gold_output: str # human-written reference rubric: list[str] # bullets the answer must address async def run_eval(agent, cases, judge_model="claude-sonnet-4-6"): results = [] for c in cases: out = await agent.run(c.input) score = await llm_judge( model=judge_model, input=c.input, gold=c.gold_output, rubric=c.rubric, actual=out, ) results.append({"id": c.id, "score": score.value, "reasoning": score.reasoning, "out": out}) return results ``` ## What to measure | Dimension | Common metric | |-----------|---------------| | Quality | LLM-judge score on rubric (0–5) | | Latency | p50 / p95 end-to-end seconds | | Cost | $ per task (tokens × pricing) | | Safety | Refusal rate on adversarial set | | Coverage | % of cases producing *some* answer | ## What not to measure - **Lines of code.** Negative correlation with quality at this stage - **Benchmark numbers transferred from papers.** Only relevant for the benchmark's task - **Tokens used per turn.** Easy to optimize; rarely matters; can come at the expense of quality ## Pre-commit eval gates Some capstones add an eval gate to their git pre-commit hook: refuse to merge if the eval drops more than 2 points. This is the highest-leverage discipline in any of the past well-scored capstones — even when the gate is informal. ## A practical pitfall Building the eval set is *more boring* than building the agent. So people skip it. **It will determine your grade.** Allocate at least 25% of your time to eval design and execution. Sources - [Anthropic — evaluating LLM applications](https://docs.claude.com/en/docs/test-and-evaluate/develop-tests) - [Ragas — open-source eval framework](https://github.com/explodinggradients/ragas) - Week 7 (Evaluation & LLM-as-a-Judge) — the long version of this lecture --- # Estimate Before You Build Capstones rarely fail because the model is too weak. They sometimes fail because the architecture chosen costs $5 per request and the demo budget allows for $0.05. ## The cost-of-a-task calculation For any agentic system, total cost per task ≈ `Σ (input_tokens × $in + output_tokens × $out) × n_calls` For a typical Pattern-2 coding agent: - 30k input tokens per turn (system + tools + context + history) - 1k output tokens per turn - 5 turns to finish a task - Sonnet 4.6 ($3 in / $15 out) ⇒ `5 × (30000 × 3e-6 + 1000 × 15e-6) = 5 × (0.09 + 0.015) = $0.53 / task` ```mermaid flowchart LR Task --> Turn1[Turn 1: 30k in, 1k out] Turn1 --> Turn2[Turn 2: 30k in, 1k out] Turn2 --> Turn3[...] Turn3 --> Final[Total: $0.53] Final --> Q{Volume?} Q -->|1000/day| D1["$530/day
= $16k/month"] Q -->|10/day| D2["$5.30/day
= $160/month"] ``` ## Where the cost concentrates - **Long agentic loops** — N turns of high context blows the budget. Compaction (lecture 11) or smaller models for sub-tasks help - **High output volumes** — `$15/M output` × 10k tokens = $0.15 per call. Use structured outputs to bound - **Re-embedding on retrieval** — embedding the query is cheap; re-embedding the entire corpus per change is expensive - **GraphRAG indexing** — one-time but real. See lecture 6 ## Two cost-reducing patterns ### 1. Tiered models ```mermaid flowchart LR Q --> Cls["Haiku 4.5
($1/M)
classify"] Cls -->|simple| Cheap[Haiku answers] Cls -->|hard| Sonnet["Sonnet 4.6
($3/M)"] Sonnet -->|stuck| Opus["Opus 4.7
($5/M)"] ``` 90% of traffic served by the cheapest tier; only the hard 10% escalates. ### 2. Aggressive prompt caching Anthropic's API caches identical system prompts at ~10% of normal cost on cache reads. A 5k-token system prompt re-used across 1000 requests saves ~$13.50. See [prompt caching docs](https://docs.claude.com/en/docs/build-with-claude/prompt-caching). ## What to include in your writeup A small table: | Component | Per-task cost | Notes | |-----------|---------------|-------| | Retrieval embedding | $0.001 | text-embedding-3-large | | Agent loop (avg 5 turns Sonnet 4.6) | $0.53 | | | Final summarization (Haiku 4.5) | $0.005 | | | **Total** | **~$0.54** | At 1000 tasks/day: $16k/month | Sources - [Anthropic — pricing](https://www.anthropic.com/pricing) - [Anthropic — prompt caching](https://docs.claude.com/en/docs/build-with-claude/prompt-caching) --- # Before You Demo A short, opinionated checklist. Hitting all of these doesn't make your system safe; failing any of them makes it definitely unsafe. ## Inputs - [ ] User input is **never** concatenated into the system prompt - [ ] Tool results are treated as **untrusted text** for prompt-injection purposes - [ ] Long-running tool outputs are bounded (max bytes returned) - [ ] Personally identifiable information is redacted from logs by default ## Actions - [ ] Every destructive tool has a confirmation step (lecture 8, permission policies) - [ ] Outbound writes (email, Slack, PR creation) are dry-run capable - [ ] No tool runs SQL without a read-only enforcement layer - [ ] Idempotency keys on any external writes that could be retried ## Models - [ ] You explicitly check `stop_reason == "refusal"` and handle it (lecture 14) - [ ] Adversarial inputs from your eval set don't produce destructive outputs - [ ] If you use multiple models, you don't pass model output from one model into another's tool call without sanitization ## Operations - [ ] You have an emergency stop (kill switch / circuit breaker) for the agent loop - [ ] You can roll back deployed prompt changes without redeploying code - [ ] You log every tool call with timestamp, args summary, result summary - [ ] You can identify which model version produced which output, ex post ```mermaid flowchart LR U[User input] --> San[Sanitize / quote] San --> Loop[Agent loop] Loop --> Tool[Tool call] Tool --> Guard{Destructive?} Guard -->|yes| Confirm[Confirm with user] Guard -->|no| Run Confirm -->|approve| Run Confirm -->|deny| Skip Run --> Result Result --> Sanitize[Sanitize tool result] Sanitize --> Loop Result --> Log[Audit log] ``` ## Two things to skip if you don't have time - **Building your own jailbreak eval.** It's a research project. Use [Anthropic's safety eval prompts](https://docs.claude.com/en/docs/test-and-evaluate/strengthen-guardrails) or [HarmBench](https://github.com/centerforaisafety/HarmBench) as a baseline - **Custom content moderation.** Llama-Guard or a hosted classifier (Anthropic, OpenAI moderation API) is good enough for capstone scope ## What to *not* skip - Permission policies on tools with side effects - Logging - A clearly-documented emergency stop Sources - [Anthropic — safety best practices](https://docs.claude.com/en/docs/test-and-evaluate/strengthen-guardrails) - [Llama-Guard 3](https://huggingface.co/meta-llama/Llama-Guard-3-8B) - [HarmBench](https://github.com/centerforaisafety/HarmBench) - Week 14 (Safety & Red-Teaming) — the long version --- # Showing the System Off 15 minutes is tight. The goal is to show what the system *does* well and to confront honestly where it fails. ## A working structure | Time | Section | What | |------|---------|------| | 0:00 – 1:00 | Setup | One slide: who you are, what the problem is, why it's worth solving | | 1:00 – 3:00 | Architecture | One diagram. Walk through the data path | | 3:00 – 9:00 | Demo | **Live or recorded run** of three scenarios (more on this) | | 9:00 – 12:00 | Evaluation | Your numbers. Where it works, where it doesn't | | 12:00 – 14:00 | Failure analysis | The hard cases. What you'd fix | | 14:00 – 15:00 | Q&A primer | Two things you anticipate being asked | ## Show three scenarios, not one - **The happy path** — proves it works at all - **A non-trivial case** — shows it goes beyond the demo script - **A failure case you didn't fix** — shows you understand the limits Skipping the third is the most common scoring miss. Showing it builds credibility. ## Live vs recorded demo Live: more impressive, riskier. Practice it three times before class. Recorded: safer, less impressive. Cut judiciously — never edit out failures, only edit out latency A recorded demo with live narration is one option. Label recordings clearly and preserve failure cases. ## What kills demos - Live coding during the presentation (don't) - Reading the system prompt out loud (the audience doesn't care; show the *behavior*) - Spending 8 minutes on architecture and 5 on the demo (inverse the proportion) - Live API keys on screen - Showing credentials, private records, or sensitive command history in a shared terminal ## A small detail that matters Have a backup. If the live demo fails, you should have a 30-second recorded clip of *the same scenario working* that you can switch to without breaking flow. Explain when a recorded backup was captured and which scenario it demonstrates. Sources - [Public capstone guide](/capstone/). Private student recordings are not included in the public materials. --- # The Section That Distinguishes Good From Great The capstone writeup has a section called "Failure Analysis." It is graded heavily. Past capstones that scored A vs B+ usually diverged on this section, not on the system itself. ## What it isn't - A list of features you didn't ship - An apology for not solving everything - A discussion of LLM limitations in general ## What it is - A **categorization** of the failure modes you observed - **Concrete examples** of each, with input → output → why - An honest assessment of which are fixable with engineering and which require fundamentally different choices - An assessment of how *frequent* each failure mode is, in your real eval ```mermaid flowchart TB Observe["Run eval → observe failures"] --> Tag[Tag each by category] Tag --> Cat1["Retrieval miss
(didn't find relevant context)"] Tag --> Cat2["Tool error
(wrong arg / wrong tool)"] Tag --> Cat3["Reasoning error
(had context, drew wrong conclusion)"] Tag --> Cat4["Hallucination
(invented fact)"] Tag --> Cat5["Refusal
(model declined to act)"] Cat1 & Cat2 & Cat3 & Cat4 & Cat5 --> Count[Count frequency
per category] Count --> Plan["What would
fix each?"] ``` ## A worked example > *"23 of 50 evaluation cases produced unsatisfactory outputs. Categorized:* > - *12 retrieval misses (the right document existed in the corpus but ranked outside top-10). Fixable with a better reranker — tried `voyage-rerank-2` on a subset and recovered 8 of 12.* > - *6 reasoning errors (correct context retrieved; model concluded wrong). 4 of these were one-step multi-hop; might benefit from explicit chain-of-thought scaffolding.* > - *3 tool errors (model called `update_record` with stale ID after a long context). Could fix with stricter tool schema or explicit state-refresh tool.* > - *2 refusals on edge-case inputs that the safety guardrail flagged as risky but were benign in context."* That paragraph is worth ~5% of the capstone grade. ## How to gather the data 1. Run your eval set 2. For every case scored < 3 / 5, classify by hand into one of your categories 3. Pick 1–2 representative examples per category for the writeup 4. **Don't paraphrase** — quote the actual inputs and outputs ## Two things to avoid - **"It works most of the time."** That's not analysis. Measure "most". - **Pretending all failures are model failures.** Most aren't — most are pipeline failures (chunking, retrieval ranking, prompt design). Be specific Sources - See `docs/capstone-rubric.md` (failure-analysis dimension) --- # Common Capstone Mistakes ## Scope mistakes - **Choosing too broad a problem.** "Build an AI doctor" can't be evaluated. "Diagnose three categories of patient intake forms from sample data" can - **Choosing a problem with no available data.** If you can't get a corpus or labeled examples by week 12, you can't evaluate. Switch - **Choosing a problem already solved by a single API call.** "Summarize a PDF" — Claude does this in one call. Your system has to add value beyond that ## Engineering mistakes - **Building before architecting.** A 30-minute architecture sketch saves weeks - **Optimizing the wrong dimension.** People micro-tune prompts for marginal quality gains while their retrieval misses 40% of the cases - **Reinventing what an MCP server already does.** Assess existing integrations before implementing another one; test their behavior and access controls ## Eval mistakes - **Skipping the eval set.** Without evaluation data, claims about quality cannot be assessed - **Evaluating on the same data the system was tuned on.** Hold-out split is non-negotiable - **LLM-judge contamination.** If your judge model is the same as your generator, scores inflate. Use a different family ## Demo mistakes - **No live demo.** "Here are some screenshots of when it worked" — not enough - **Long architecture lecture, short demo.** Audience wants to see the thing *do something* - **No failure case shown.** Include a failure scenario so reviewers can assess the limits ## Writeup mistakes - **No numbers.** Every claim ("worked well", "fast", "cheap") needs a number - **No failure analysis.** See previous slide - **Architecture diagram with everything labeled "LLM."** Specify which model, which framework, which version - **Demo screenshots without the failure column.** Always show both ## One meta-mistake Treating the capstone as **a presentation about an AI system** rather than **an AI system that you happen to present**. The system is the deliverable. The presentation is the proof that the system exists and works. ```mermaid flowchart LR Right["Build → measure →
iterate → demo"] -->|graded| A["A range"] Wrong["Plan → demo →
build a thin demo →
write a lot"] -->|graded| BC["B / C range"] ``` Sources - `docs/capstone-rubric.md`, especially the "deliverable weight" table --- # How You'll Be Scored | Dimension | Weight | Looks like | |-----------|-------:|------------| | Problem clarity | 10% | Defined scope; "could plausibly ship"; not solved by one API call | | Architectural thinking | 15% | Diagram; explicit choices; trade-offs articulated | | Engineering quality | 15% | Runs from clean clone; sensible error handling; readable code | | Evaluation rigor | 20% | Eval set built early; numbers for quality + cost + latency; honest about scope | | Failure analysis | 15% | Categorized, quantified, fixable-vs-fundamental distinction | | Demo execution | 10% | Three scenarios; one failure shown; live or seamless recording | | Writeup quality | 10% | 4–6 pages; diagrams; concrete; reads like an internal design doc | | Capstone ingredients used | 5% | At least three from the required list (RAG, MCP, multi-agent, ...) | ## What an A looks like - Runs end-to-end on first try from a clean clone - Has 50+ evaluation cases with measured quality + cost + latency - Failure analysis names 3–5 specific failure modes with frequencies - Demo includes a scenario that doesn't work, explained - Writeup reads like an engineering design doc, not a school assignment ## What a B looks like - Runs but with manual setup steps - Has 10–20 eval cases or vague qualitative eval - Failure analysis is general (one paragraph, no specific cases) - Demo is happy-path only - Writeup describes what was built but not the alternatives considered ## What a C looks like - Doesn't run end-to-end without instructor intervention - No eval set - No failure analysis or it amounts to "the LLM sometimes gets things wrong" - Demo is screenshots only - Writeup is a feature list ## Soft expectations not in the rubric - **Citations.** Don't pretend you invented techniques you didn't. Cite the papers and tools you used - **Reproducibility.** Pin model versions and library versions. Include seeds where applicable - **Cost transparency.** Per-task cost belongs in your writeup, even if your numbers are small ## What to prioritize - The system being **smaller** than you originally planned. Shipping a tight, well-evaluated subset beats a sprawling, broken whole - The system using **existing components**. Building on `microsoft/graphrag`, `LangGraph`, an MCP server you didn't write — all fine, as long as you explain why and measure your additions - The model choice being **a smaller model**. Evaluate whether it meets the task requirements before choosing a larger one Sources - [Public capstone guide](/capstone/) uses the table in this slide as its source. See `docs/capstone-rubric.md` for the publication arrangement. --- # Plan the Capstone These milestones describe a suggested self-study sequence. Choose your own dates. The public edition has no submission service, staffed office hours, or instructor grading. | Course point | Milestone | |------|-----------| | Week 10 | Draft a one-page proposal: problem, architecture, model candidates, evaluation plan | | Week 12 | Collect baseline results on at least 10 cases | | Week 14 | Stabilize the demonstration and document remaining failures | | Week 15 | Complete the demonstration, writeup, code, and evaluation results | ## Artifacts to prepare - **Code:** a Git repository with dependencies and setup instructions another person can follow - **Writeup:** 4–6 pages with architecture, alternatives, measured results, and limits - **Demonstration:** up to 15 minutes showing normal behavior, a difficult case, and a failure - **Evaluation:** CSV or JSON with per-case results and a description of the scoring method ## Resources actually available The public repository provides lecture Markdown and the slide viewer. Classroom starter projects, sample databases, private recordings, and student submissions are not included. For a new implementation, use official documentation and disposable data sources; see the [resources page](/resources/). ## Review questions - Can someone reproduce the result from your instructions? - Are the evaluation data separate from the examples used during development? - Which failures have you observed, and which paths remain untested? - Can a reviewer understand your model choice and system boundaries? ```mermaid flowchart LR Proposal[Define scope and evidence] --> Baseline[Build a baseline] Baseline --> Evaluate[Measure and revise] Evaluate --> Demo[Demonstrate behavior] Demo --> Review[Document results and limits] ``` Sources - [Public capstone guide](/capstone/) --- # Keep Building and Measuring ## What to take away A working mental model of how to build, evaluate, and operate AI systems. Tools and model lineups change; the discipline of defining a task, measuring behavior, and explaining failures stays useful. ## Three habits to keep 1. **Measure before optimizing.** Establish a task metric and a baseline. 2. **Make boundaries explicit.** Define who can authorize an action and what happens when a dependency fails. 3. **Report the evidence.** Record quality, latency, cost, and the limits of what you tested. ## Continue your study - [Course schedule](/schedule/) - [Capstone guide](/capstone/) - [Edition and version notes](/updates/) - [Contact the instructor](https://sawanruparel.com/contact) The public edition is self-study. Current university offering details and assessment arrangements belong to the relevant university course.