Week 1: The AI model landscape

Slide 1 of 30Open this slide

The 2026 AI Landscape

From Stochastic Parrots to Reasoning Machines

Applied AI Course -- Spring 2026

Week 1: Setting the Stage

Slide 2 of 30Open this slide

Key Milestones: 2023-2025

Mar 2023 -- GPT-4 launches: multimodal, passes the bar exam (90th percentile)
Nov 2023 -- GPT-4 Turbo: 128K context, cheaper API pricing
Dec 2023 -- Mixtral 8x7B: open-weight MoE proves efficient alternatives exist
Mar 2024 -- Claude 3 Opus: competitive with GPT-4, strong on analysis and safety
Apr 2024 -- Llama 3 (8B, 70B): Meta's open-weight push accelerates the ecosystem
Jun 2024 -- Claude 3.5 Sonnet: faster than Opus, matches or exceeds it on benchmarks
Sep 2024 -- OpenAI o1: first commercial "reasoning model" with chain-of-thought at inference
Jan 2025 -- DeepSeek R1: open-weight reasoning model trained with RL, rivaling o1
2025 -- o3, Claude 4, Gemini 2.0: reasoning and tool use become table stakes

The pace of change is not slowing down -- it is accelerating and shifting direction.

Sources

Slide 3 of 30Open this slide

The Paradigm Shift: Bigger Models vs. Smarter Inference

Old Paradigm (2020-2023)

Scale model parameters: 175B, 540B, 1T+
More training data = better performance
Chinchilla scaling laws dominate strategy
Single forward pass per query
"Emergent abilities" from scale alone
Cost paid once at training time

New Paradigm (2024-2026)

Allocate more compute at inference time
Let the model "think" before answering
Chain-of-thought, backtracking, verification
Variable compute per query complexity
Reasoning as a learnable skill (via RL)
Cost scales with problem difficulty

flowchart LR subgraph Old["Old paradigm (2020-2023)"] D1[More data] --> L1[Larger model] --> P1[One forward pass] --> A1[Answer] end subgraph New["New paradigm (2024-2026)"] Q2[Query] --> L2[Smaller-or-equal model] L2 --> R[Reasoning loop] R --> R R --> A2[Answer] end

The frontier moved from "how big is your model?" to "how well does your model think?"

Sources

Slide 4 of 30Open this slide

Why This Matters for Practitioners

Model selection is now multi-dimensional -- cost, latency, reasoning depth, and context length all trade off against each other
Prompt engineering is evolving -- reasoning models need less hand-holding but benefit from structured problem decomposition
Architecture choices changed -- compound AI systems with multiple models beat a single monolithic call
Open-weight models are competitive -- DeepSeek R1, Llama 3, Qwen 2.5 mean you can self-host reasoning
Evaluation is harder -- benchmarks are saturating; you need domain-specific evals
Cost profiles are shifting -- a 30-second o3 call costs more than 100 GPT-4o calls, but may be worth it

The skill gap is no longer "can you call an API?" -- it is "can you design systems that use these capabilities effectively?"

Sources

Slide 5 of 30Open this slide

Kahneman's Framework Applied to LLMs

System 1: Fast Thinking

Automatic, effortless, intuitive
Pattern matching from experience
"What is 2 + 2?" -- instant
Prone to cognitive biases
LLM analogy: single forward pass, next-token prediction, pattern completion
Good enough for most everyday tasks

System 2: Slow Thinking

Deliberate, effortful, logical
Step-by-step reasoning
"What is 347 x 283?" -- requires work
Can override System 1 errors
LLM analogy: chain-of-thought, self-verification, search over reasoning paths
Essential for complex, novel problems

flowchart LR Q[Question] --> Router{Easy or hard?} Router -- easy --> S1["System 1 (fast, intuitive)"] Router -- hard --> S2["System 2 (slow, deliberate)"] S1 --> Ans[Answer] S2 --> Ans

Inference-time scaling gives LLMs their first real "System 2" -- the ability to think harder when the problem demands it.

Sources

Thinking, Fast and Slow (Daniel Kahneman, 2011)

Slide 6 of 30Open this slide

Traditional LLMs as System 1 Thinkers

Single forward pass: input tokens in, output tokens out -- no deliberation
Fixed compute per token: every token gets the same amount of processing regardless of difficulty
Pattern completion: trained to predict the next token, not to "reason"
Fast and fluent: produces grammatical, coherent text almost instantly
Confidently wrong: generates plausible-sounding answers even when the answer is incorrect

Where System 1 LLMs excel:

Summarization, translation, creative writing, code completion
Tasks where pattern matching from training data is sufficient
High-throughput, low-latency applications

Where System 1 LLMs fail:

Multi-step math (GSM8K: GPT-3 ~10%, GPT-4 ~92% -- but only with CoT prompting)
Logic puzzles, formal reasoning, constraint satisfaction
Planning and search problems
Novel problems not well-represented in training data

Sources

Slide 7 of 30Open this slide

Inference-Time Compute: System 2 for LLMs

Core techniques enabling "slow thinking":

Chain-of-Thought (CoT): model generates intermediate reasoning steps before the final answer
Self-verification: model checks its own work, catches errors, and revises
Backtracking: model explores one path, recognizes a dead end, and tries another approach
Internal search: model considers multiple solution paths (tree-of-thoughts)
Reflection: model critiques its own reasoning and improves iteratively

What changed with o1/R1:

CoT is no longer just a prompting trick -- it is trained into the model via RL
Models learn when to think harder and how to allocate reasoning effort
Reasoning tokens are generated internally (hidden from the user in o1, visible in R1)
Performance scales with inference-time compute: more thinking = better answers

The key insight: you can trade compute at inference time for accuracy, just like you trade compute at training time for capability.

Sources

Slide 8 of 30Open this slide

Training-Time vs. Inference-Time Scaling

flowchart LR subgraph Train["Training-time scaling (Chinchilla)"] T1[Small model] --> T2[Medium model] --> T3[Large model diminishing returns] end subgraph Infer["Inference-time scaling (o1, o3, R1)"] I1[Direct answer] --> I2[Short CoT] --> I3[Long CoT + search + verify] end Train -.compose.-> Best[Larger model + more thinking = best results] Infer -.compose.-> Best

Key observations:

Training-time scaling (Chinchilla laws): diminishing returns as models get larger
Inference-time scaling (o1/o3/R1): new dimension of improvement, orthogonal to model size
They compose: a larger model + more inference compute = best results
Practical implication: a smaller model thinking longer can beat a larger model answering instantly
OpenAI reported o3 outperforming o1 by spending 10-100x more inference compute on hard problems

Newer scaling laws (2026):

Train-to-Test (T2) scaling laws: it is compute-optimal to train smaller models on more data and then spend the saved budget on inference-time sampling -- shifting compute from training toward test time
Kinetics Scaling Law: adds KV-cache bandwidth as an explicit cost variable, recognizing that inference-time scaling is gated by memory bandwidth, not just FLOPs

Sources

Slide 9 of 30Open this slide

How Inference-Time Compute Works: Think Longer, Not Bigger

The mechanism:

Problem arrives -- model assesses difficulty
Reasoning tokens generated -- internal chain-of-thought (can be 1K-100K+ tokens)
Search and backtracking -- model explores paths, prunes bad ones
Self-verification -- model checks consistency, re-derives results
Final answer produced -- only after deliberation is complete

flowchart LR P[Problem] --> A[Assess difficulty] A --> R[Generate reasoning tokens] R --> S{Path looks good?} S -- no --> R S -- yes --> V[Self-verify] V --> D{Consistent?} D -- no --> R D -- yes --> F[Final answer]

Concrete numbers:

Simple factual question: ~50 reasoning tokens, ~0.5s
Math competition problem: ~10,000 reasoning tokens, ~30s
ARC-AGI hard task (o3): ~100,000+ reasoning tokens, ~minutes
Cost scales linearly with reasoning tokens generated

Why reinforcement learning matters:

Models are trained with RL (e.g., GRPO in DeepSeek R1) to produce useful reasoning
Reward signal: did the final answer turn out correct?
The model learns to allocate effort proportional to problem difficulty
Without RL, chain-of-thought is often superficial or repetitive

Sources

Slide 10 of 30Open this slide

OpenAI o-Series: Reasoning Models

o1 (September 2024)

First commercial reasoning model with trained chain-of-thought
Hidden "thinking tokens" -- user sees only the final answer
AIME 2024 math: 83.3% (vs GPT-4o at 13.4%)
GPQA Diamond (PhD-level science): 78.0%
Latency: 10-60 seconds per response for hard problems

o1-pro (Late 2024)

Higher compute budget per query, available via ChatGPT Pro ($200/mo)
Improved reliability on hardest tasks

o3 (2025)

Significant jump: ARC-AGI benchmark 87.5% (high-compute), vs o1 at 25%
Configurable compute: low/medium/high reasoning effort
Cost at high compute: ~$1000+ per ARC-AGI task
SWE-bench Verified: substantial improvements in real-world coding

GPT-5.4 (March 2026) and GPT-5.5 (April 23, 2026)

GPT-5.4 is tuned for agentic execution -- reliable long-horizon tool use and multi-step task completion
GPT-5.5 introduces a 1-million-token context window and tops the abstract-reasoning leaderboard
GPT-5.5 posts 85% on ARC-AGI-2 (as of May 13, 2026), the new de-facto benchmark for abstract reasoning
Together they mark the point where context length and inference-time reasoning both jumped a generation

o-series proved that inference-time scaling is a real, reliable axis of improvement -- not a parlor trick.

Sources

Slide 11 of 30Open this slide

DeepSeek R1: Open-Weight Reasoning

Why R1 matters:

Open weights -- full model available for download and self-hosting
Competitive with o1 on math, coding, and science benchmarks
Transparent reasoning -- chain-of-thought is visible, not hidden
Trained by a Chinese AI lab (DeepSeek) at a fraction of the rumored cost

Technical approach:

Base model: DeepSeek-V3 (MoE, 671B total params, ~37B active)
GRPO (Group Relative Policy Optimization): RL without a separate critic model
Key finding: RL alone (without supervised fine-tuning on CoT data) produces emergent reasoning
The model spontaneously learned to self-verify, backtrack, and allocate variable effort

Benchmark highlights:

AIME 2024: 79.8% (pass@1), comparable to o1 at 79.2%
MATH-500: 97.3%
Codeforces: 96.3 percentile rating
Distilled versions (1.5B-70B) retain strong reasoning ability

R1 democratized reasoning models: you no longer need a $200/mo subscription to access System 2 thinking.

Sources

Slide 12 of 30Open this slide

Claude: From 3 to 4

Claude 3 family (March 2024)

Haiku (fast/cheap), Sonnet (balanced), Opus (best quality)
First to seriously challenge GPT-4 across benchmarks
Strong instruction following, safety, and long-context performance

Claude 3.5 Sonnet (June 2024)

Matched or exceeded Opus quality at Sonnet speed and cost
SWE-bench Verified: 49% (state-of-the-art at the time)
Introduced computer use (beta): model can operate a desktop GUI

Claude 4 (2025)

Extended thinking: explicit chain-of-thought reasoning (visible to developers)
Up to 128K+ thinking tokens on hard problems
Strong tool use: multi-step function calling with planning
Computer use GA: autonomous browser and desktop interaction
200K context window standard

Claude Opus 4.7 (April 16, 2026)

Leads SWE-Bench Verified at 80.8% -- the strongest real-world coding model on the benchmark
Positioned for long-horizon reasoning tasks: deep multi-step analysis and agentic coding
Continues the extended-thinking lineage with developer-controllable reasoning budgets

Distinctive strengths:

Careful instruction following and nuanced safety behavior
Strong at long-document analysis and structured outputs
Extended thinking is developer-controllable (budget tokens for reasoning)

Sources

Slide 13 of 30Open this slide

Google's Approach: Gemini 2.0

Gemini 2.0 Flash Thinking

Google's answer to reasoning models
"Thinking" mode that shows reasoning process
Optimized for speed: reasoning at Flash-tier latency
Strong on math and science benchmarks

Long context as an alternative strategy

Gemini 1.5 Pro: 1M token context window (later 2M)
Gemini 2.0: maintained long context with improved quality
Key insight: long context can replace RAG for many use cases
Stuff the entire codebase/document set into context instead of retrieval

Gemini 3.1 Pro (February 2026)

Leads GPQA Diamond at 94.3% -- the strongest score on PhD-level science reasoning
The only natively multimodal frontier model spanning text, image, audio, and video in one model
Pairs frontier reasoning with Gemini's large-context heritage (1M+ tokens)

Multimodal native

Trained natively on text, images, audio, and video
Video understanding: process hours of footage directly
Strong on visual reasoning tasks (charts, diagrams, screenshots)

Practical considerations:

Competitive API pricing, especially Flash tier
Deep integration with Google Cloud, Vertex AI
Grounding with Google Search built in
NotebookLM: showcase of long-context + multimodal

Sources

Slide 14 of 30Open this slide

The Open-Weight Landscape

Llama 4 (Meta)

Shipped April 2026 as a Mixture-of-Experts family
Scout: 17B active / 109B total params; Maverick: 17B active / 400B total params
Sparse MoE keeps inference cheap while raising total capacity
Massive ecosystem: fine-tunes, quantizations, tooling
Foundation for thousands of derivative models

Qwen 3.5 / 3.6 (Alibaba)

Qwen 3.5 (397B) and Qwen 3.6 (27B) shipped March–May 2026 under Apache 2.0
Qwen 3.6 hits 77.2% on SWE-bench despite its compact 27B size
Strong multilingual performance (especially CJK)
Coder-specialized variants remain a standout strength
Competitive with the top open-weight models at equivalent sizes

Mistral / Mixtral

Mixtral 8x22B: MoE architecture, efficient inference
Mistral Large: closed-weight flagship
Strong European AI alternative
Pioneered efficient MoE for open models

DeepSeek (open-weight reasoning)

DeepSeek V3.2 (January 2026): outperforms GPT-5 on several reasoning tasks while staying open-weight
Keeps frontier-class reasoning fully downloadable and self-hostable
DeepSeek R2 (anticipated 1.2T-param MoE, ~78B active) has not yet shipped as of May 2026
Continues the lineage that made open reasoning models mainstream

Phi (Microsoft)

Phi-3/Phi-4: "small language models" (3.8B-14B)
Trained on high-quality synthetic + curated data
Punches well above weight class on reasoning
Ideal for edge deployment and fine-tuning

Sources

Slide 15 of 30Open this slide

Model Comparison (Approximate, Early 2026)

Model	Input $/1M tok	Output $/1M tok	Context	Reasoning	Speed
GPT-4o	$2.50	$10.00	128K	Standard	Fast
GPT-5.4 (Mar 2026)	varies	varies	256K	Strong (agentic)	Fast
GPT-5.5 (Apr 2026)	varies	varies	1M	Very strong	Medium
o3 (high)	$10.00	$40.00	128K	Very strong	Slow
Claude 4 Opus	$15.00	$75.00	200K	Extended thinking	Medium
Claude Opus 4.7 (Apr 2026)	varies	varies	200K	Extended thinking	Medium
Gemini 2.0 Flash	$0.10	$0.40	1M	Flash thinking	Very fast
Gemini 3.1 Pro (Feb 2026)	varies	varies	1M+	Standard (multimodal)	Fast
DeepSeek R1	$0.55	$2.19	64K	Strong (visible CoT)	Medium
Llama 4 Maverick*	Self-host	Self-host	1M	Standard (MoE)	Varies

Self-hosted costs depend on infrastructure. Prices are approximate and change frequently.

There is no single "best" model -- the right choice depends on your task, budget, and latency requirements.

Sources

Slide 16 of 30Open this slide

Decision Framework: Choosing the Right Model

flowchart TB T[Define task] --> C{Complexity?} C -- simple --> Fast["Fast model (GPT-4o, Sonnet, Flash)"] C -- multi-step --> Reason["Reasoning model (o3, R1, ext. thinking)"] C -- huge context --> Long["Long-context (Gemini 2M, Claude 200K)"] Fast --> Con{Constraints?} Reason --> Con Long --> Con Con -- privacy --> Open[Open-weight self-host] Con -- ok --> Proto[Prototype + evaluate] Open --> Proto

Step 1: Define the Task

Classification, generation, reasoning, code, multimodal?
What accuracy threshold is acceptable?
Is the task well-represented in training data, or novel?

Step 2: Assess Complexity

Simple pattern matching -> standard model (GPT-4o, Sonnet, Flash)
Multi-step reasoning required -> reasoning model (o3, R1, extended thinking)
Massive context needed -> long-context model (Gemini 2M, Claude 200K)

Step 3: Check Constraints

Latency: real-time (<1s) vs. async (seconds-minutes acceptable)?
Cost: high-volume (need cheap) vs. low-volume (can afford premium)?
Privacy: can data leave your infrastructure? If not -> open-weight models
Compliance: regulatory requirements for model provenance?

Step 4: Prototype and Evaluate

Start with the cheapest model that might work
Build domain-specific evals before committing
Test with real data, not benchmarks
Consider compound systems: cheap model for easy tasks, expensive for hard ones

Slide 17 of 30Open this slide

LLMs Are Not Chatbots

The chatbot era is over. LLMs are now:

Reasoning engines -- solve complex problems with multi-step logic
Code interpreters -- write, execute, and debug code autonomously
Tool orchestrators -- call APIs, query databases, browse the web
Autonomous agents -- plan, act, observe, and iterate toward goals
Pipeline components -- one step in a larger software system

Real-world deployment patterns in 2026:

Coding assistants: Cursor, GitHub Copilot, Claude Code -- not chatbots, but IDE-integrated agents
Research agents: deep web research, paper analysis, competitive intelligence
Data pipelines: LLMs as transformation steps (extract, classify, summarize, route)
Customer support: multi-turn resolution with tool access (not scripted FAQ bots)
DevOps: automated incident response, log analysis, root cause identification

If your mental model of LLMs is "fancy autocomplete" or "better chatbot," you are thinking too small.

Sources

Slide 18 of 30Open this slide

Compound AI Systems

flowchart TB Orc["Orchestrator (Router / Agent)"] Orc --> Fast["Fast model (GPT-4o, Flash)"] Orc --> Reason["Reasoning model (o3, R1, extended)"] Orc --> Spec["Specialized (code, vision)"] Fast --> Vec[Vector DB / RAG] Reason --> Sand[Code sandbox] Spec --> API[External APIs]

Key principles:

Route by difficulty: cheap model for easy queries, expensive model for hard ones
Route across vendors at runtime: production teams treat model selection as a runtime decision, routing to GPT-5.4 for agentic execution, Claude Opus 4.7 for long reasoning, and Gemini 3.1 Pro for multimodal / large-context work
Compose capabilities: combine reasoning + retrieval + code execution
Verify outputs: use one model to check another's work
The system is the product, not any single model call

Sources

Slide 19 of 30Open this slide

Inference-Time Patterns: Trading Compute for Accuracy

Reasoning models don't just answer -- they can think in structured ways. The core patterns at a glance:

Chain-of-Thought (CoT): generate reasoning steps before the answer -- now trained natively, not just prompted
Self-Consistency: sample multiple CoT paths, take the majority vote -- trades compute for reliability
Tree-of-Thoughts (ToT): explore and prune multiple reasoning branches
Reflection / self-critique: the model evaluates and revises its own output -- the foundation of agent loops (act -> observe -> reflect -> act)

These are the building blocks of "thinking longer." We go deep on each -- when to use them, how to prompt them, and where they fail -- in Week 2 (Advanced Prompt Engineering).

Sources

Slide 20 of 30Open this slide

Tool Use: LLMs That Act in the World

Why tool use changes everything:

LLMs alone are limited to knowledge frozen at training time
Tools give models access to live data, computation, and external systems
Tool use + reasoning = agents that can solve real problems autonomously

Core tool categories:

Search & retrieval: web search, vector DB queries, document lookup
Code execution: Python sandboxes, shell commands, REPL environments
API calls: REST/GraphQL endpoints, SaaS integrations, database queries
File operations: read, write, edit files in a codebase or filesystem
Browser/computer use: navigate websites, fill forms, operate GUIs

How it works in practice:

Model receives a task and a list of available tools (function schemas)
Model decides which tool to call and with what arguments
Tool executes, returns results to the model
Model reasons about results, decides next action or produces final answer

sequenceDiagram autonumber participant U as User participant M as Model participant T as Tool U->>M: Task + tool schemas M->>M: Decide which tool to call M->>T: Call tool(args) T-->>M: Result M->>M: Reason: done or call another? alt more work M->>T: Call tool(args) T-->>M: Result end M-->>U: Final answer

State of the art:

Claude: parallel tool use, multi-step tool chains, computer use
GPT-4o/o3: function calling with structured outputs
Gemini: grounded generation with Google Search
Open models catching up via fine-tuning on tool-use datasets

Slide 21 of 30Open this slide

The Context Window Revolution

The progression:

GPT-3 (2020): 4K tokens (~3,000 words)
GPT-4 (2023): 8K-32K tokens
GPT-4 Turbo (2023): 128K tokens (~100,000 words)
Claude 3 (2024): 200K tokens (~150,000 words)
Gemini 1.5 Pro (2024): 1M tokens, later 2M (~1.5M words)

What 1M+ tokens means in practice:

An entire codebase (10,000+ lines) in a single prompt
A full book or research paper collection
Hours of meeting transcripts
Hundreds of pages of legal documents

Implications for system design:

RAG becomes optional for many use cases -- just stuff it in context
Few-shot learning with dozens or hundreds of examples
Long-form analysis without chunking and reassembly
"Needle in a haystack" retrieval within the context window itself

Limitations to remember:

Cost scales linearly with context length
Attention to information in the "middle" can degrade (lost-in-the-middle effect)
Latency increases with context size
Not all models handle long context equally well -- test empirically

Sources

Slide 22 of 30Open this slide

Multimodality: Not Just Text Anymore

Vision

All frontier models now accept images natively
Chart/diagram understanding, OCR, screenshot analysis
UI understanding enables computer use agents
Medical imaging, satellite imagery, document processing

Audio

Speech-to-text and text-to-speech integrated into models
Gemini: native audio understanding (tone, music, ambient sound)
GPT-4o: real-time voice conversation mode
Whisper/open alternatives for self-hosted speech processing

Video

Gemini 2.0: process hours of video directly
Frame-by-frame analysis without external tools
Surveillance, content moderation, sports analysis
Still early: high cost, limited availability

Code & Structured Data

Models understand and generate code in 50+ languages
JSON/XML/YAML structured outputs with schema validation
SQL generation from natural language
Direct interaction with databases and APIs

Sources

Slide 23 of 30Open this slide

The Hallucination Problem

Reasoning helps but does not solve it:

Standard LLMs hallucinate ~15-25% of factual claims (varies by domain and model)
Reasoning models reduce hallucination through self-verification, but still fail
o1/R1 can produce "confidently wrong" reasoning chains -- the logic looks sound but premises are fabricated

Why hallucinations persist:

Models generate plausible continuations, not verified facts
Training data contains errors, contradictions, and outdated information
Reasoning over wrong premises produces wrong conclusions -- with high confidence
No built-in "I don't know" mechanism (though instruction tuning helps)

Mitigation strategies (we will cover these in depth):

Retrieval-Augmented Generation (RAG): ground responses in retrieved documents
Citations and attribution: force models to cite sources, verify against them
Tool use: let the model look things up rather than relying on parametric memory
Self-consistency: sample multiple times, flag disagreements
Human-in-the-loop: design systems where humans verify critical outputs

flowchart LR Q[Query] --> M[LLM] M --> H{Risk of hallucination?} H -- high --> RAG[Ground via RAG] H -- high --> Tools[Tools / search] H -- high --> SC[Self-consistency] RAG --> V[Verify + cite] Tools --> V SC --> V V --> HIL{Critical output?} HIL -- yes --> Human[Human review] HIL -- no --> Out[Answer] Human --> Out

Hallucination is not a bug that will be patched -- it is a fundamental property of generative models that must be engineered around.

Sources

Slide 24 of 30Open this slide

Cost and Latency Tradeoffs

Cheap & Fast (System 1)

GPT-4o mini: ~$0.15/$0.60 per 1M tokens
Gemini 2.0 Flash: ~$0.10/$0.40 per 1M tokens
Claude 3.5 Haiku: ~$0.80/$4.00 per 1M tokens
Latency: 100-500ms for typical responses
Use for: classification, extraction, summarization, simple Q&A, routing
1M API calls/day is financially viable

Expensive & Slow (System 2)

o3 (high compute): ~$10/$40 per 1M tokens + reasoning tokens
Claude 4 Opus w/ extended thinking: ~$15/$75 per 1M tokens
Reasoning tokens can be 10-100x the output length
Latency: 10s-120s for complex problems
Use for: math, code generation, complex analysis, planning, research
1M calls/day would cost $10,000-$100,000+

flowchart LR Q[Incoming query] --> R{Difficulty} R -- simple --> S1["System 1 (Flash, Haiku, 4o-mini)"] R -- hard --> S2["System 2 (o3, Opus + thinking)"] S1 --> A[Answer] S2 --> A

The winning architecture in 2026: route easy tasks to cheap models, hard tasks to reasoning models. Pay for thinking only when you need it.

Sources

Slide 25 of 30Open this slide

The Evaluation Crisis

Benchmarks are saturating:

MMLU: GPT-4 86.4% (2023) -> frontier models ~90%+ (2025) -- ceiling effect
HumanEval (code): solved by most frontier models at 90%+
GSM8K (math): went from challenging to trivial in 18 months
ARC-AGI-2: released as a harder successor to ARC-AGI to combat saturation -- human average is 66%, and it is now the de-facto leaderboard for abstract reasoning; GPT-5.5 leads at 85%
When models ace the test, the test stops being informative

Leaderboards are unreliable:

Data contamination: benchmark questions leak into training data
Overfitting to benchmarks: models (or their training) optimized for specific tests
Chatbot Arena (LMSYS): crowdsourced preference, but biased toward style over substance
Self-reported results from labs are marketing, not science

What to do instead:

Build domain-specific evals for your actual use case
Use held-out test sets that are not public
Test on real user queries, not synthetic benchmarks
Evaluate systems, not models -- the full pipeline matters
LLM-as-judge: use a strong model to evaluate a weaker one (but watch for biases)
Track regression: eval after every model update or prompt change

If you are choosing a model based on a leaderboard, you are probably making the wrong decision.

Sources

Slide 26 of 30Open this slide

Safety and Alignment: A Preview

The landscape of risks:

Prompt injection: adversarial inputs that hijack model behavior
Jailbreaking: bypassing safety training to produce harmful outputs
Data poisoning: compromised training data leading to backdoors
Misuse: generating disinformation, malware, social engineering content
Autonomy risks: agents taking unintended actions in the real world

Current mitigation approaches:

RLHF / Constitutional AI: training models to be helpful, harmless, and honest
System prompts & guardrails: runtime constraints on model behavior
Red-teaming: adversarial testing before deployment
Content filtering: input/output classifiers that catch harmful content
Sandboxing: limiting what agents can access and execute

Why reasoning models add new dimensions:

Longer reasoning chains = more surface area for manipulation
Autonomous agents can take irreversible real-world actions
Tool use means models interact with external systems
"Thinking" tokens may contain reasoning the developer cannot inspect (o1)

We will dedicate Week 12 to safety, red-teaming, and responsible deployment practices.

Sources

Slide 27 of 30Open this slide

Course Roadmap: 15 Weeks

Weeks 1-3: Foundations

Week 1: The 2026 AI Landscape (today)
Week 2: Prompt Engineering & Structured Outputs
Week 3: APIs, SDKs, and Model Integration

Weeks 4-6: Core Patterns

Week 4: Retrieval-Augmented Generation (RAG)
Week 5: Function Calling & Tool Use
Week 6: Agents and Autonomous Systems

Weeks 7-9: Building Systems

Week 7: Evaluation and Testing
Week 8: Compound AI Systems & Orchestration
Week 9: Fine-Tuning and Adaptation

Weeks 10-12: Production & Safety

Week 10: Deployment, Scaling, and Cost Optimization
Week 11: Multimodal Applications
Week 12: Safety, Red-Teaming, and Responsible AI

Weeks 13-15: Capstone

Week 13: Advanced Topics (inference-time compute, open models)
Week 14: Capstone Project Work
Week 15: Capstone Presentations

Slide 28 of 30Open this slide

Key Takeaways

The field shifted

The frontier moved from "bigger models" to "smarter inference." Inference-time compute (o3, R1, extended thinking) is the most important development since the transformer.

No single best model

Model selection is now a multi-dimensional optimization problem: cost, latency, reasoning depth, context length, privacy, and deployment constraints all matter.

Systems over models

The winning approach is compound AI systems -- routing, orchestration, tool use, and verification -- not a single monolithic model call.

Evaluation is your moat

Public benchmarks are saturating and unreliable. Your ability to build domain-specific evaluations is what separates production systems from demos.

Open-weight models are real

DeepSeek R1, Llama 3, Qwen 2.5, and others make self-hosted, private, customizable AI systems practical for the first time.

The skills gap is changing

The bottleneck is no longer API access -- it is the ability to design, evaluate, and deploy systems that use these capabilities effectively.

Slide 29 of 30Open this slide

Key Papers and Readings

Paper / Resource	Year	Why It Matters
Attention Is All You Need (Vaswani et al.)	2017	The transformer architecture that started it all
Scaling Laws for Neural Language Models (Kaplan et al.)	2020	Established training-time scaling laws
Chain-of-Thought Prompting (Wei et al.)	2022	Showed reasoning emerges from step-by-step prompting
Training Compute-Optimal LLMs (Hoffmann et al., Chinchilla)	2022	Optimal balance of model size and training data
GPT-4 Technical Report (OpenAI)	2023	Frontier multimodal model capabilities
Scaling LLM Test-Time Compute (Snell et al.)	2024	Formal framework for inference-time scaling
DeepSeek-R1 Technical Report	2025	Open-weight reasoning via RL, emergent CoT
OpenAI o1 System Card	2024	Safety analysis of reasoning models
The Shift from Models to Compound AI Systems (Zaharia et al.)	2024	Why systems beat individual models

Week 1 Complete

Next week: Prompt Engineering & Structured Outputs

Before next class:

Read the DeepSeek-R1 technical report (Section 3: RL approach)
Set up API access for at least two providers (OpenAI + one other)
Try the same complex prompt on GPT-4o, o3, Claude 4, and DeepSeek R1 -- compare the outputs

Questions?