The project
Build a working AI system for a clearly scoped problem. Establish what success means before choosing a model or architecture. Keep the scope small enough to build, run, and evaluate end to end.
The guide adapts the existing Week 15 project criteria for independent study. Use the rubric for self-assessment or peer review; the public edition does not provide instructor grading.
Deliverables
- A working application: a Git repository with pinned dependencies and setup instructions that someone else can follow.
- A demonstration: up to 15 minutes, live or recorded, covering a normal case, a challenging case, and a failure case.
- An architecture writeup: 4–6 pages describing the problem, data flow, alternatives, selected design, and operational limits.
- Evaluation results: a held-out set, baseline comparison, and measured quality, cost, and latency. Include machine-readable case results.
- Failure analysis: concrete examples, categorized failure modes, observed frequencies, and proposed improvements.
Technical ingredients
The original project asks you to combine at least three of these ingredients:
- Retrieval through vector RAG, GraphRAG, or hybrid search.
- An agentic loop with tool use.
- A custom MCP server or a substantial MCP integration.
- Multi-agent orchestration.
- Human approval points or persistent memory.
- Fine-tuning, distillation, or routing across model sizes.
- A safety layer with evaluation coverage.
Select ingredients because the problem calls for them. Compare the result with a simpler baseline and explain the cost of added complexity.
Suggested milestones
| Course point | Evidence to produce |
|---|---|
| Week 10 | One-page proposal: problem, scope, architecture, model candidates, and evaluation plan. |
| Week 12 | Initial evaluation on at least 10 cases, with baseline quality, cost, and latency. |
| Week 14 | A stable demonstration build and a documented list of remaining failures. |
| Week 15 | Reproducible code, demonstration, writeup, and final evaluation results. |
For self-study, these are sequencing suggestions. There are no public submission deadlines or staffed office hours.
Self-assessment rubric
The following weights are reproduced from the existing Week 15 rubric. They sum to 100% of the capstone assessment.
| Dimension | Weight | Looks like |
|---|---|---|
| Problem clarity | 10% | Defined scope; "could plausibly ship"; not solved by one API call |
| Architectural thinking | 15% | Diagram; explicit choices; trade-offs articulated |
| Engineering quality | 15% | Runs from clean clone; sensible error handling; readable code |
| Evaluation rigor | 20% | Eval set built early; numbers for quality + cost + latency; honest about scope |
| Failure analysis | 15% | Categorized, quantified, fixable-vs-fundamental distinction |
| Demo execution | 10% | Three scenarios; one failure shown; live or seamless recording |
| Writeup quality | 10% | 4–6 pages; diagrams; concrete; reads like an internal design doc |
| Capstone ingredients used | 5% | At least three from the required list (RAG, MCP, multi-agent, ...) |
Demonstrating the system
Spend most of the demonstration showing actual behavior and evidence. Give a short problem introduction, walk through one architecture diagram, run the three scenarios, and explain evaluation results and failures. Keep credentials and personal data off screen. If the live system is unavailable, label a recording as a recording and explain what was tested.
Reproducibility and safety
Record your model versions, dataset provenance, package versions, environment requirements, and experiment dates. Test external actions against disposable resources. Include rejected requests and recovery behavior in your evaluation. State which results you actually measured and which remain unverified.
Related lecture
Week 15: Capstone develops the project patterns, evaluation strategy, cost model, safety review, and failure analysis in more detail. The resources page links to primary implementation documentation.