# Week 1: The AI model landscape Applied AI Course — Sawan Ruparel Public self-study edition. See https://applied-ai.sawanruparel.com/updates/ for version notes. # The 2026 AI Landscape ## From Stochastic Parrots to Reasoning Machines **Applied AI Course -- Spring 2026** Week 1: Setting the Stage --- # Key Milestones: 2023-2025 - **Mar 2023** -- GPT-4 launches: multimodal, passes the bar exam (90th percentile) - **Nov 2023** -- GPT-4 Turbo: 128K context, cheaper API pricing - **Dec 2023** -- Mixtral 8x7B: open-weight MoE proves efficient alternatives exist - **Mar 2024** -- Claude 3 Opus: competitive with GPT-4, strong on analysis and safety - **Apr 2024** -- Llama 3 (8B, 70B): Meta's open-weight push accelerates the ecosystem - **Jun 2024** -- Claude 3.5 Sonnet: faster than Opus, matches or exceeds it on benchmarks - **Sep 2024** -- OpenAI o1: first commercial "reasoning model" with chain-of-thought at inference - **Jan 2025** -- DeepSeek R1: open-weight reasoning model trained with RL, rivaling o1 - **2025** -- o3, Claude 4, Gemini 2.0: reasoning and tool use become table stakes > The pace of change is not slowing down -- it is accelerating and shifting direction. ## Sources - [GPT-4 Technical Report (OpenAI, 2023)](https://arxiv.org/abs/2303.08774) - [New Models and Developer Products at DevDay (OpenAI, Nov 2023)](https://openai.com/index/new-models-and-developer-products-announced-at-devday/) - [Mixtral of Experts (Mistral AI, Dec 2023)](https://mistral.ai/news/mixtral-of-experts) - [Introducing Claude 3 (Anthropic, Mar 2024)](https://www.anthropic.com/news/claude-3-family) - [Introducing Meta Llama 3 (Meta, Apr 2024)](https://ai.meta.com/blog/meta-llama-3/) - [Introducing Claude 3.5 Sonnet (Anthropic, Jun 2024)](https://www.anthropic.com/news/claude-3-5-sonnet) - [Introducing OpenAI o1 (OpenAI, Sep 2024)](https://openai.com/index/introducing-openai-o1-preview/) - [DeepSeek-R1 Release (DeepSeek, Jan 2025)](https://github.com/deepseek-ai/DeepSeek-R1) - [Introducing Claude 4 (Anthropic, 2025)](https://www.anthropic.com/news/claude-4) - [Introducing Gemini 2.0 (Google, 2024)](https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/) --- # The Paradigm Shift: Bigger Models vs. Smarter Inference ## Old Paradigm (2020-2023) - Scale model parameters: 175B, 540B, 1T+ - More training data = better performance - Chinchilla scaling laws dominate strategy - Single forward pass per query - "Emergent abilities" from scale alone - Cost paid once at training time ## New Paradigm (2024-2026) - Allocate more compute at inference time - Let the model "think" before answering - Chain-of-thought, backtracking, verification - Variable compute per query complexity - Reasoning as a learnable skill (via RL) - Cost scales with problem difficulty ```mermaid flowchart LR subgraph Old["Old paradigm (2020-2023)"] D1[More data] --> L1[Larger model] --> P1[One forward pass] --> A1[Answer] end subgraph New["New paradigm (2024-2026)"] Q2[Query] --> L2[Smaller-or-equal model] L2 --> R[Reasoning loop] R --> R R --> A2[Answer] end ``` > The frontier moved from "how big is your model?" to "how well does your model think?" ## Sources - [Training Compute-Optimal Large Language Models — Chinchilla (Hoffmann et al., 2022)](https://arxiv.org/abs/2203.15556) - [Chain-of-Thought Prompting Elicits Reasoning in LLMs (Wei et al., 2022)](https://arxiv.org/abs/2201.11903) - [DeepSeek-R1: Incentivizing Reasoning via RL (DeepSeek, 2025)](https://arxiv.org/abs/2501.12948) - [OpenAI o1 System Card (OpenAI, 2024)](https://arxiv.org/abs/2412.16720) - [Scaling LLM Test-Time Compute Optimally (Snell et al., 2024)](https://arxiv.org/abs/2408.03314) --- # Why This Matters for Practitioners - **Model selection is now multi-dimensional** -- cost, latency, reasoning depth, and context length all trade off against each other - **Prompt engineering is evolving** -- reasoning models need less hand-holding but benefit from structured problem decomposition - **Architecture choices changed** -- compound AI systems with multiple models beat a single monolithic call - **Open-weight models are competitive** -- DeepSeek R1, Llama 3, Qwen 2.5 mean you can self-host reasoning - **Evaluation is harder** -- benchmarks are saturating; you need domain-specific evals - **Cost profiles are shifting** -- a 30-second o3 call costs more than 100 GPT-4o calls, but may be worth it > The skill gap is no longer "can you call an API?" -- it is "can you design systems that use these capabilities effectively?" ## Sources - [The Shift from Models to Compound AI Systems (Zaharia et al., 2024)](https://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/) - [DeepSeek-R1 (DeepSeek, 2025)](https://arxiv.org/abs/2501.12948) - [Introducing Meta Llama 3 (Meta, 2024)](https://ai.meta.com/blog/meta-llama-3/) - [Qwen2.5 (Alibaba, 2024)](https://qwenlm.github.io/blog/qwen2.5/) - [OpenAI API Pricing](https://openai.com/api/pricing/) --- # Kahneman's Framework Applied to LLMs ## System 1: Fast Thinking - Automatic, effortless, intuitive - Pattern matching from experience - "What is 2 + 2?" -- instant - Prone to cognitive biases - **LLM analogy:** single forward pass, next-token prediction, pattern completion - Good enough for most everyday tasks ## System 2: Slow Thinking - Deliberate, effortful, logical - Step-by-step reasoning - "What is 347 x 283?" -- requires work - Can override System 1 errors - **LLM analogy:** chain-of-thought, self-verification, search over reasoning paths - Essential for complex, novel problems ```mermaid flowchart LR Q[Question] --> Router{Easy or hard?} Router -- easy --> S1["System 1
(fast, intuitive)"] Router -- hard --> S2["System 2
(slow, deliberate)"] S1 --> Ans[Answer] S2 --> Ans ``` > Inference-time scaling gives LLMs their first real "System 2" -- the ability to think harder when the problem demands it. ## Sources - [Thinking, Fast and Slow (Daniel Kahneman, 2011)](https://us.macmillan.com/books/9780374533557/thinkingfastandslow) --- # Traditional LLMs as System 1 Thinkers - **Single forward pass:** input tokens in, output tokens out -- no deliberation - **Fixed compute per token:** every token gets the same amount of processing regardless of difficulty - **Pattern completion:** trained to predict the next token, not to "reason" - **Fast and fluent:** produces grammatical, coherent text almost instantly - **Confidently wrong:** generates plausible-sounding answers even when the answer is incorrect ### Where System 1 LLMs excel: - Summarization, translation, creative writing, code completion - Tasks where pattern matching from training data is sufficient - High-throughput, low-latency applications ### Where System 1 LLMs fail: - Multi-step math (GSM8K: GPT-3 ~10%, GPT-4 ~92% -- but only with CoT prompting) - Logic puzzles, formal reasoning, constraint satisfaction - Planning and search problems - Novel problems not well-represented in training data ## Sources - [Training Verifiers to Solve Math Word Problems — GSM8K (Cobbe et al., 2021)](https://arxiv.org/abs/2110.14168) - [GPT-4 Technical Report (OpenAI, 2023)](https://arxiv.org/abs/2303.08774) - [Chain-of-Thought Prompting Elicits Reasoning in LLMs (Wei et al., 2022)](https://arxiv.org/abs/2201.11903) --- # Inference-Time Compute: System 2 for LLMs ### Core techniques enabling "slow thinking": - **Chain-of-Thought (CoT):** model generates intermediate reasoning steps before the final answer - **Self-verification:** model checks its own work, catches errors, and revises - **Backtracking:** model explores one path, recognizes a dead end, and tries another approach - **Internal search:** model considers multiple solution paths (tree-of-thoughts) - **Reflection:** model critiques its own reasoning and improves iteratively ### What changed with o1/R1: - CoT is no longer just a prompting trick -- it is **trained into the model via RL** - Models learn *when* to think harder and *how* to allocate reasoning effort - Reasoning tokens are generated internally (hidden from the user in o1, visible in R1) - Performance scales with inference-time compute: more thinking = better answers > The key insight: you can trade compute at inference time for accuracy, just like you trade compute at training time for capability. ## Sources - [Chain-of-Thought Prompting Elicits Reasoning in LLMs (Wei et al., 2022)](https://arxiv.org/abs/2201.11903) - [Tree of Thoughts: Deliberate Problem Solving with LLMs (Yao et al., 2023)](https://arxiv.org/abs/2305.10601) - [Reflexion: Language Agents with Verbal Reinforcement Learning (Shinn et al., 2023)](https://arxiv.org/abs/2303.11366) - [OpenAI o1 System Card (OpenAI, 2024)](https://arxiv.org/abs/2412.16720) - [DeepSeek-R1 (DeepSeek, 2025)](https://arxiv.org/abs/2501.12948) --- # Training-Time vs. Inference-Time Scaling ```mermaid flowchart LR subgraph Train["Training-time scaling (Chinchilla)"] T1[Small model] --> T2[Medium model] --> T3[Large model
diminishing returns] end subgraph Infer["Inference-time scaling (o1, o3, R1)"] I1[Direct answer] --> I2[Short CoT] --> I3[Long CoT
+ search + verify] end Train -.compose.-> Best[Larger model + more thinking = best results] Infer -.compose.-> Best ``` ### Key observations: - **Training-time scaling** (Chinchilla laws): diminishing returns as models get larger - **Inference-time scaling** (o1/o3/R1): new dimension of improvement, orthogonal to model size - **They compose:** a larger model + more inference compute = best results - **Practical implication:** a smaller model thinking longer can beat a larger model answering instantly - OpenAI reported o3 outperforming o1 by spending 10-100x more inference compute on hard problems ### Newer scaling laws (2026): - **Train-to-Test (T2) scaling laws:** it is compute-optimal to train *smaller* models on more data and then spend the saved budget on inference-time sampling -- shifting compute from training toward test time - **Kinetics Scaling Law:** adds **KV-cache bandwidth** as an explicit cost variable, recognizing that inference-time scaling is gated by memory bandwidth, not just FLOPs ## Sources - [Training Compute-Optimal Large Language Models — Chinchilla (Hoffmann et al., 2022)](https://arxiv.org/abs/2203.15556) - [Scaling LLM Test-Time Compute Optimally (Snell et al., 2024)](https://arxiv.org/abs/2408.03314) - [Train-to-Test (T2) and Kinetics Scaling Laws (2026)](https://arxiv.org/html/2604.01411v1) - [OpenAI o1 System Card (OpenAI, 2024)](https://arxiv.org/abs/2412.16720) --- # How Inference-Time Compute Works: Think Longer, Not Bigger ### The mechanism: 1. **Problem arrives** -- model assesses difficulty 2. **Reasoning tokens generated** -- internal chain-of-thought (can be 1K-100K+ tokens) 3. **Search and backtracking** -- model explores paths, prunes bad ones 4. **Self-verification** -- model checks consistency, re-derives results 5. **Final answer produced** -- only after deliberation is complete ```mermaid flowchart LR P[Problem] --> A[Assess difficulty] A --> R[Generate reasoning tokens] R --> S{Path looks good?} S -- no --> R S -- yes --> V[Self-verify] V --> D{Consistent?} D -- no --> R D -- yes --> F[Final answer] ``` ### Concrete numbers: - Simple factual question: ~50 reasoning tokens, ~0.5s - Math competition problem: ~10,000 reasoning tokens, ~30s - ARC-AGI hard task (o3): ~100,000+ reasoning tokens, ~minutes - Cost scales linearly with reasoning tokens generated ### Why reinforcement learning matters: - Models are trained with RL (e.g., GRPO in DeepSeek R1) to produce *useful* reasoning - Reward signal: did the final answer turn out correct? - The model learns to allocate effort proportional to problem difficulty - Without RL, chain-of-thought is often superficial or repetitive ## Sources - [DeepSeek-R1: Incentivizing Reasoning via RL (DeepSeek, 2025)](https://arxiv.org/abs/2501.12948) - [Scaling LLM Test-Time Compute Optimally (Snell et al., 2024)](https://arxiv.org/abs/2408.03314) - [OpenAI o1 System Card (OpenAI, 2024)](https://arxiv.org/abs/2412.16720) --- # OpenAI o-Series: Reasoning Models ### o1 (September 2024) - First commercial reasoning model with trained chain-of-thought - Hidden "thinking tokens" -- user sees only the final answer - AIME 2024 math: 83.3% (vs GPT-4o at 13.4%) - GPQA Diamond (PhD-level science): 78.0% - Latency: 10-60 seconds per response for hard problems ### o1-pro (Late 2024) - Higher compute budget per query, available via ChatGPT Pro ($200/mo) - Improved reliability on hardest tasks ### o3 (2025) - Significant jump: ARC-AGI benchmark 87.5% (high-compute), vs o1 at 25% - Configurable compute: low/medium/high reasoning effort - Cost at high compute: ~$1000+ per ARC-AGI task - SWE-bench Verified: substantial improvements in real-world coding ### GPT-5.4 (March 2026) and GPT-5.5 (April 23, 2026) - GPT-5.4 is tuned for **agentic execution** -- reliable long-horizon tool use and multi-step task completion - GPT-5.5 introduces a **1-million-token context window** and tops the abstract-reasoning leaderboard - GPT-5.5 posts **85% on ARC-AGI-2** (as of May 13, 2026), the new de-facto benchmark for abstract reasoning - Together they mark the point where context length and inference-time reasoning both jumped a generation > o-series proved that inference-time scaling is a real, reliable axis of improvement -- not a parlor trick. ## Sources - [Introducing OpenAI o1 (OpenAI, Sep 2024)](https://openai.com/index/introducing-openai-o1-preview/) - [OpenAI o1 System Card (OpenAI, 2024)](https://arxiv.org/abs/2412.16720) - [Introducing GPT-5.5 (OpenAI, 2026)](https://openai.com/index/introducing-gpt-5-5/) - [GPT-5.5 Review for Developers and Builders (MindStudio, 2026)](https://www.mindstudio.ai/blog/gpt-5-5-review-developers-builders) - [OpenAI API Pricing](https://openai.com/api/pricing/) --- # DeepSeek R1: Open-Weight Reasoning ### Why R1 matters: - **Open weights** -- full model available for download and self-hosting - **Competitive with o1** on math, coding, and science benchmarks - **Transparent reasoning** -- chain-of-thought is visible, not hidden - Trained by a Chinese AI lab (DeepSeek) at a fraction of the rumored cost ### Technical approach: - Base model: DeepSeek-V3 (MoE, 671B total params, ~37B active) - **GRPO (Group Relative Policy Optimization):** RL without a separate critic model - Key finding: RL alone (without supervised fine-tuning on CoT data) produces emergent reasoning - The model spontaneously learned to self-verify, backtrack, and allocate variable effort ### Benchmark highlights: - AIME 2024: 79.8% (pass@1), comparable to o1 at 79.2% - MATH-500: 97.3% - Codeforces: 96.3 percentile rating - Distilled versions (1.5B-70B) retain strong reasoning ability > R1 democratized reasoning models: you no longer need a $200/mo subscription to access System 2 thinking. ## Sources - [DeepSeek-R1: Incentivizing Reasoning via RL (DeepSeek, 2025)](https://arxiv.org/abs/2501.12948) - [DeepSeek-R1 GitHub Repository](https://github.com/deepseek-ai/DeepSeek-R1) - [DeepSeek-R1 Release Announcement](https://api-docs.deepseek.com/news/news250120) --- # Claude: From 3 to 4 ### Claude 3 family (March 2024) - Haiku (fast/cheap), Sonnet (balanced), Opus (best quality) - First to seriously challenge GPT-4 across benchmarks - Strong instruction following, safety, and long-context performance ### Claude 3.5 Sonnet (June 2024) - Matched or exceeded Opus quality at Sonnet speed and cost - SWE-bench Verified: 49% (state-of-the-art at the time) - Introduced **computer use** (beta): model can operate a desktop GUI ### Claude 4 (2025) - **Extended thinking:** explicit chain-of-thought reasoning (visible to developers) - Up to 128K+ thinking tokens on hard problems - Strong tool use: multi-step function calling with planning - Computer use GA: autonomous browser and desktop interaction - 200K context window standard ### Claude Opus 4.7 (April 16, 2026) - **Leads SWE-Bench Verified at 80.8%** -- the strongest real-world coding model on the benchmark - Positioned for **long-horizon reasoning** tasks: deep multi-step analysis and agentic coding - Continues the extended-thinking lineage with developer-controllable reasoning budgets ### Distinctive strengths: - Careful instruction following and nuanced safety behavior - Strong at long-document analysis and structured outputs - Extended thinking is developer-controllable (budget tokens for reasoning) ## Sources - [Introducing Claude 3 Family (Anthropic, Mar 2024)](https://www.anthropic.com/news/claude-3-family) - [Introducing Claude 3.5 Sonnet (Anthropic, Jun 2024)](https://www.anthropic.com/news/claude-3-5-sonnet) - [Introducing Claude 4 (Anthropic, 2025)](https://www.anthropic.com/news/claude-4) - [GPT-5.5 Review: Claude Opus 4.7 and Gemini 3.1 Pro Comparison (MindStudio, 2026)](https://www.mindstudio.ai/blog/gpt-5-5-review-developers-builders) - [Anthropic API Pricing](https://platform.claude.com/docs/en/about-claude/pricing) --- # Google's Approach: Gemini 2.0 ### Gemini 2.0 Flash Thinking - Google's answer to reasoning models - "Thinking" mode that shows reasoning process - Optimized for speed: reasoning at Flash-tier latency - Strong on math and science benchmarks ### Long context as an alternative strategy - Gemini 1.5 Pro: 1M token context window (later 2M) - Gemini 2.0: maintained long context with improved quality - **Key insight:** long context can replace RAG for many use cases - Stuff the entire codebase/document set into context instead of retrieval ### Gemini 3.1 Pro (February 2026) - **Leads GPQA Diamond at 94.3%** -- the strongest score on PhD-level science reasoning - The **only natively multimodal frontier model** spanning text, image, audio, and video in one model - Pairs frontier reasoning with Gemini's large-context heritage (1M+ tokens) ### Multimodal native - Trained natively on text, images, audio, and video - Video understanding: process hours of footage directly - Strong on visual reasoning tasks (charts, diagrams, screenshots) ### Practical considerations: - Competitive API pricing, especially Flash tier - Deep integration with Google Cloud, Vertex AI - Grounding with Google Search built in - NotebookLM: showcase of long-context + multimodal ## Sources - [Our Next-Generation Model: Gemini 1.5 (Google, Feb 2024)](https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/) - [Introducing Gemini 2.0 (Google, Dec 2024)](https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/) - [GPT-5.5 Review: Gemini 3.1 Pro Multimodal Comparison (MindStudio, 2026)](https://www.mindstudio.ai/blog/gpt-5-5-review-developers-builders) - [Google NotebookLM](https://notebooklm.google/) - [Google AI Gemini API Pricing](https://ai.google.dev/gemini-api/docs/pricing) --- # The Open-Weight Landscape ## Llama 4 (Meta) - Shipped **April 2026** as a Mixture-of-Experts family - **Scout:** 17B active / 109B total params; **Maverick:** 17B active / 400B total params - Sparse MoE keeps inference cheap while raising total capacity - Massive ecosystem: fine-tunes, quantizations, tooling - Foundation for thousands of derivative models ## Qwen 3.5 / 3.6 (Alibaba) - **Qwen 3.5 (397B)** and **Qwen 3.6 (27B)** shipped March–May 2026 under **Apache 2.0** - Qwen 3.6 hits **77.2% on SWE-bench** despite its compact 27B size - Strong multilingual performance (especially CJK) - Coder-specialized variants remain a standout strength - Competitive with the top open-weight models at equivalent sizes ## Mistral / Mixtral - Mixtral 8x22B: MoE architecture, efficient inference - Mistral Large: closed-weight flagship - Strong European AI alternative - Pioneered efficient MoE for open models ## DeepSeek (open-weight reasoning) - **DeepSeek V3.2 (January 2026):** outperforms GPT-5 on several reasoning tasks while staying open-weight - Keeps frontier-class reasoning fully downloadable and self-hostable - **DeepSeek R2** (anticipated 1.2T-param MoE, ~78B active) has **not yet shipped** as of May 2026 - Continues the lineage that made open reasoning models mainstream ## Phi (Microsoft) - Phi-3/Phi-4: "small language models" (3.8B-14B) - Trained on high-quality synthetic + curated data - Punches well above weight class on reasoning - Ideal for edge deployment and fine-tuning ## Sources - [Best Open-Source LLMs 2026: Llama 4, Qwen 3.5, DeepSeek V4, Gemma 4, Mistral (Codersera, 2026)](https://codersera.com/blog/best-open-source-llm-2026-llama-4-qwen-3-5-deepseek-v4-gemma-4-mistral/) - [DeepSeek V3.2 Outperforms GPT-5 on Reasoning Tasks (InfoQ, Jan 2026)](https://www.infoq.com/news/2026/01/deepseek-v32/) - [Mixtral of Experts (Mistral AI, Dec 2023)](https://mistral.ai/news/mixtral-of-experts) - [Introducing Phi-3: Redefining What's Possible with SLMs (Microsoft, Apr 2024)](https://azure.microsoft.com/en-us/blog/introducing-phi-3-redefining-whats-possible-with-slms/) - [Introducing Phi-4 (Microsoft, Dec 2024)](https://techcommunity.microsoft.com/blog/azure-ai-foundry-blog/introducing-phi-4-microsoft%E2%80%99s-newest-small-language-model-specializing-in-comple/4357090) --- # Model Comparison (Approximate, Early 2026) | Model | Input $/1M tok | Output $/1M tok | Context | Reasoning | Speed | |-------|----------------|-----------------|---------|-----------|-------| | GPT-4o | $2.50 | $10.00 | 128K | Standard | Fast | | GPT-5.4 (Mar 2026) | varies | varies | 256K | Strong (agentic) | Fast | | GPT-5.5 (Apr 2026) | varies | varies | 1M | Very strong | Medium | | o3 (high) | $10.00 | $40.00 | 128K | Very strong | Slow | | Claude 4 Opus | $15.00 | $75.00 | 200K | Extended thinking | Medium | | Claude Opus 4.7 (Apr 2026) | varies | varies | 200K | Extended thinking | Medium | | Gemini 2.0 Flash | $0.10 | $0.40 | 1M | Flash thinking | Very fast | | Gemini 3.1 Pro (Feb 2026) | varies | varies | 1M+ | Standard (multimodal) | Fast | | DeepSeek R1 | $0.55 | $2.19 | 64K | Strong (visible CoT) | Medium | | Llama 4 Maverick* | Self-host | Self-host | 1M | Standard (MoE) | Varies | *Self-hosted costs depend on infrastructure. Prices are approximate and change frequently.* > There is no single "best" model -- the right choice depends on your task, budget, and latency requirements. ## Sources - [OpenAI API Pricing](https://openai.com/api/pricing/) - [Anthropic API Pricing](https://platform.claude.com/docs/en/about-claude/pricing) - [Google AI Gemini API Pricing](https://ai.google.dev/gemini-api/docs/pricing) - [DeepSeek API Pricing](https://api-docs.deepseek.com/news/news250120) - [Introducing Meta Llama 3 (Meta, 2024)](https://ai.meta.com/blog/meta-llama-3/) --- # Decision Framework: Choosing the Right Model ```mermaid flowchart TB T[Define task] --> C{Complexity?} C -- simple --> Fast["Fast model
(GPT-4o, Sonnet, Flash)"] C -- multi-step --> Reason["Reasoning model
(o3, R1, ext. thinking)"] C -- huge context --> Long["Long-context
(Gemini 2M, Claude 200K)"] Fast --> Con{Constraints?} Reason --> Con Long --> Con Con -- privacy --> Open[Open-weight self-host] Con -- ok --> Proto[Prototype + evaluate] Open --> Proto ``` ## Step 1: Define the Task - Classification, generation, reasoning, code, multimodal? - What accuracy threshold is acceptable? - Is the task well-represented in training data, or novel? ## Step 2: Assess Complexity - Simple pattern matching -> standard model (GPT-4o, Sonnet, Flash) - Multi-step reasoning required -> reasoning model (o3, R1, extended thinking) - Massive context needed -> long-context model (Gemini 2M, Claude 200K) ## Step 3: Check Constraints - **Latency:** real-time (<1s) vs. async (seconds-minutes acceptable)? - **Cost:** high-volume (need cheap) vs. low-volume (can afford premium)? - **Privacy:** can data leave your infrastructure? If not -> open-weight models - **Compliance:** regulatory requirements for model provenance? ## Step 4: Prototype and Evaluate - Start with the cheapest model that might work - Build domain-specific evals before committing - Test with real data, not benchmarks - Consider compound systems: cheap model for easy tasks, expensive for hard ones --- # LLMs Are Not Chatbots ### The chatbot era is over. LLMs are now: - **Reasoning engines** -- solve complex problems with multi-step logic - **Code interpreters** -- write, execute, and debug code autonomously - **Tool orchestrators** -- call APIs, query databases, browse the web - **Autonomous agents** -- plan, act, observe, and iterate toward goals - **Pipeline components** -- one step in a larger software system ### Real-world deployment patterns in 2026: - **Coding assistants:** Cursor, GitHub Copilot, Claude Code -- not chatbots, but IDE-integrated agents - **Research agents:** deep web research, paper analysis, competitive intelligence - **Data pipelines:** LLMs as transformation steps (extract, classify, summarize, route) - **Customer support:** multi-turn resolution with tool access (not scripted FAQ bots) - **DevOps:** automated incident response, log analysis, root cause identification > If your mental model of LLMs is "fancy autocomplete" or "better chatbot," you are thinking too small. ## Sources - [Cursor — The Best Way to Code with AI](https://cursor.com/) - [GitHub Copilot](https://github.com/features/copilot) - [Claude Code (Anthropic)](https://claude.com/product/claude-code) --- # Compound AI Systems ```mermaid flowchart TB Orc["Orchestrator
(Router / Agent)"] Orc --> Fast["Fast model
(GPT-4o, Flash)"] Orc --> Reason["Reasoning model
(o3, R1, extended)"] Orc --> Spec["Specialized
(code, vision)"] Fast --> Vec[Vector DB / RAG] Reason --> Sand[Code sandbox] Spec --> API[External APIs] ``` ### Key principles: - **Route by difficulty:** cheap model for easy queries, expensive model for hard ones - **Route across vendors at runtime:** production teams treat model selection as a *runtime decision*, routing to GPT-5.4 for agentic execution, Claude Opus 4.7 for long reasoning, and Gemini 3.1 Pro for multimodal / large-context work - **Compose capabilities:** combine reasoning + retrieval + code execution - **Verify outputs:** use one model to check another's work - **The system is the product**, not any single model call ## Sources - [The Shift from Models to Compound AI Systems (Zaharia et al., 2024)](https://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/) - [Multi-Model Orchestration: GPT-5.4, Claude Opus 4.7, Gemini 3.1 (ALM Corp, 2026)](https://almcorp.com/blog/multi-model-orchestration-gpt-5-4-claude-opus-4-7-gemini-3-1/) --- # Inference-Time Patterns: Trading Compute for Accuracy Reasoning models don't just answer -- they can think in structured ways. The core patterns at a glance: - **Chain-of-Thought (CoT):** generate reasoning steps before the answer -- now trained natively, not just prompted - **Self-Consistency:** sample multiple CoT paths, take the majority vote -- trades compute for reliability - **Tree-of-Thoughts (ToT):** explore and prune multiple reasoning branches - **Reflection / self-critique:** the model evaluates and revises its own output -- the foundation of agent loops (act -> observe -> reflect -> act) > These are the building blocks of "thinking longer." We go deep on each -- when to use them, how to prompt them, and where they fail -- in **Week 2 (Advanced Prompt Engineering)**. ## Sources - [Chain-of-Thought Prompting Elicits Reasoning in LLMs (Wei et al., 2022)](https://arxiv.org/abs/2201.11903) - [Self-Consistency Improves Chain of Thought Reasoning (Wang et al., 2022)](https://arxiv.org/abs/2203.11171) - [Tree of Thoughts: Deliberate Problem Solving with LLMs (Yao et al., 2023)](https://arxiv.org/abs/2305.10601) --- # Tool Use: LLMs That Act in the World ### Why tool use changes everything: - LLMs alone are limited to knowledge frozen at training time - Tools give models access to **live data, computation, and external systems** - Tool use + reasoning = agents that can solve real problems autonomously ### Core tool categories: - **Search & retrieval:** web search, vector DB queries, document lookup - **Code execution:** Python sandboxes, shell commands, REPL environments - **API calls:** REST/GraphQL endpoints, SaaS integrations, database queries - **File operations:** read, write, edit files in a codebase or filesystem - **Browser/computer use:** navigate websites, fill forms, operate GUIs ### How it works in practice: 1. Model receives a task and a list of available tools (function schemas) 2. Model decides which tool to call and with what arguments 3. Tool executes, returns results to the model 4. Model reasons about results, decides next action or produces final answer ```mermaid sequenceDiagram autonumber participant U as User participant M as Model participant T as Tool U->>M: Task + tool schemas M->>M: Decide which tool to call M->>T: Call tool(args) T-->>M: Result M->>M: Reason: done or call another? alt more work M->>T: Call tool(args) T-->>M: Result end M-->>U: Final answer ``` ### State of the art: - Claude: parallel tool use, multi-step tool chains, computer use - GPT-4o/o3: function calling with structured outputs - Gemini: grounded generation with Google Search - Open models catching up via fine-tuning on tool-use datasets --- # The Context Window Revolution ### The progression: - GPT-3 (2020): 4K tokens (~3,000 words) - GPT-4 (2023): 8K-32K tokens - GPT-4 Turbo (2023): 128K tokens (~100,000 words) - Claude 3 (2024): 200K tokens (~150,000 words) - Gemini 1.5 Pro (2024): 1M tokens, later 2M (~1.5M words) ### What 1M+ tokens means in practice: - An entire codebase (10,000+ lines) in a single prompt - A full book or research paper collection - Hours of meeting transcripts - Hundreds of pages of legal documents ### Implications for system design: - **RAG becomes optional** for many use cases -- just stuff it in context - **Few-shot learning** with dozens or hundreds of examples - **Long-form analysis** without chunking and reassembly - **"Needle in a haystack"** retrieval within the context window itself ### Limitations to remember: - Cost scales linearly with context length - Attention to information in the "middle" can degrade (lost-in-the-middle effect) - Latency increases with context size - Not all models handle long context equally well -- test empirically ## Sources - [GPT-4 Technical Report (OpenAI, 2023)](https://arxiv.org/abs/2303.08774) - [Introducing Claude 3 Family (Anthropic, Mar 2024)](https://www.anthropic.com/news/claude-3-family) - [Our Next-Generation Model: Gemini 1.5 (Google, Feb 2024)](https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/) - [Lost in the Middle: How Language Models Use Long Contexts (Liu et al., 2023)](https://arxiv.org/abs/2307.03172) --- # Multimodality: Not Just Text Anymore ## Vision - All frontier models now accept images natively - Chart/diagram understanding, OCR, screenshot analysis - UI understanding enables computer use agents - Medical imaging, satellite imagery, document processing ## Audio - Speech-to-text and text-to-speech integrated into models - Gemini: native audio understanding (tone, music, ambient sound) - GPT-4o: real-time voice conversation mode - Whisper/open alternatives for self-hosted speech processing ## Video - Gemini 2.0: process hours of video directly - Frame-by-frame analysis without external tools - Surveillance, content moderation, sports analysis - Still early: high cost, limited availability ## Code & Structured Data - Models understand and generate code in 50+ languages - JSON/XML/YAML structured outputs with schema validation - SQL generation from natural language - Direct interaction with databases and APIs ## Sources - [Introducing Gemini 2.0 (Google, Dec 2024)](https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/) - [Introducing GPT-4o (OpenAI, May 2024)](https://openai.com/index/hello-gpt-4o/) - [OpenAI Whisper — GitHub](https://github.com/openai/whisper) --- # The Hallucination Problem ### Reasoning helps but does not solve it: - **Standard LLMs** hallucinate ~15-25% of factual claims (varies by domain and model) - **Reasoning models** reduce hallucination through self-verification, but still fail - o1/R1 can produce "confidently wrong" reasoning chains -- the logic looks sound but premises are fabricated ### Why hallucinations persist: - Models generate plausible continuations, not verified facts - Training data contains errors, contradictions, and outdated information - Reasoning over wrong premises produces wrong conclusions -- with high confidence - No built-in "I don't know" mechanism (though instruction tuning helps) ### Mitigation strategies (we will cover these in depth): - **Retrieval-Augmented Generation (RAG):** ground responses in retrieved documents - **Citations and attribution:** force models to cite sources, verify against them - **Tool use:** let the model look things up rather than relying on parametric memory - **Self-consistency:** sample multiple times, flag disagreements - **Human-in-the-loop:** design systems where humans verify critical outputs ```mermaid flowchart LR Q[Query] --> M[LLM] M --> H{Risk of hallucination?} H -- high --> RAG[Ground via RAG] H -- high --> Tools[Tools / search] H -- high --> SC[Self-consistency] RAG --> V[Verify + cite] Tools --> V SC --> V V --> HIL{Critical output?} HIL -- yes --> Human[Human review] HIL -- no --> Out[Answer] Human --> Out ``` > Hallucination is not a bug that will be patched -- it is a fundamental property of generative models that must be engineered around. ## Sources - [FActScore: Fine-grained Atomic Evaluation of Factual Precision (Min et al., 2023)](https://arxiv.org/abs/2305.14251) - [OpenAI o1 System Card (OpenAI, 2024)](https://arxiv.org/abs/2412.16720) --- # Cost and Latency Tradeoffs ## Cheap & Fast (System 1) - GPT-4o mini: ~$0.15/$0.60 per 1M tokens - Gemini 2.0 Flash: ~$0.10/$0.40 per 1M tokens - Claude 3.5 Haiku: ~$0.80/$4.00 per 1M tokens - Latency: 100-500ms for typical responses - **Use for:** classification, extraction, summarization, simple Q&A, routing - 1M API calls/day is financially viable ## Expensive & Slow (System 2) - o3 (high compute): ~$10/$40 per 1M tokens + reasoning tokens - Claude 4 Opus w/ extended thinking: ~$15/$75 per 1M tokens - Reasoning tokens can be 10-100x the output length - Latency: 10s-120s for complex problems - **Use for:** math, code generation, complex analysis, planning, research - 1M calls/day would cost $10,000-$100,000+ ```mermaid flowchart LR Q[Incoming query] --> R{Difficulty} R -- simple --> S1["System 1
(Flash, Haiku, 4o-mini)"] R -- hard --> S2["System 2
(o3, Opus + thinking)"] S1 --> A[Answer] S2 --> A ``` > The winning architecture in 2026: route easy tasks to cheap models, hard tasks to reasoning models. Pay for thinking only when you need it. ## Sources - [OpenAI API Pricing](https://openai.com/api/pricing/) - [Anthropic API Pricing](https://platform.claude.com/docs/en/about-claude/pricing) - [Google AI Gemini API Pricing](https://ai.google.dev/gemini-api/docs/pricing) --- # The Evaluation Crisis ### Benchmarks are saturating: - MMLU: GPT-4 86.4% (2023) -> frontier models ~90%+ (2025) -- ceiling effect - HumanEval (code): solved by most frontier models at 90%+ - GSM8K (math): went from challenging to trivial in 18 months - **ARC-AGI-2:** released as a harder successor to ARC-AGI to combat saturation -- human average is 66%, and it is now the de-facto leaderboard for abstract reasoning; GPT-5.5 leads at 85% - When models ace the test, the test stops being informative ### Leaderboards are unreliable: - **Data contamination:** benchmark questions leak into training data - **Overfitting to benchmarks:** models (or their training) optimized for specific tests - **Chatbot Arena** (LMSYS): crowdsourced preference, but biased toward style over substance - **Self-reported results** from labs are marketing, not science ### What to do instead: - **Build domain-specific evals** for your actual use case - **Use held-out test sets** that are not public - **Test on real user queries,** not synthetic benchmarks - **Evaluate systems, not models** -- the full pipeline matters - **LLM-as-judge:** use a strong model to evaluate a weaker one (but watch for biases) - **Track regression:** eval after every model update or prompt change > If you are choosing a model based on a leaderboard, you are probably making the wrong decision. ## Sources - [Measuring Massive Multitask Language Understanding — MMLU (Hendrycks et al., 2020)](https://arxiv.org/abs/2009.03300) - [Evaluating Large Language Models Trained on Code — HumanEval (Chen et al., 2021)](https://arxiv.org/abs/2107.03374) - [Training Verifiers to Solve Math Word Problems — GSM8K (Cobbe et al., 2021)](https://arxiv.org/abs/2110.14168) - [ARC-AGI-2 Benchmark (ARC Prize Foundation)](https://arcprize.org/arc-agi/2) - [ARC-AGI-2: A New Challenge for Frontier AI Reasoning (2025)](https://arxiv.org/abs/2505.11831) - [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023)](https://arxiv.org/abs/2306.05685) - [Chatbot Arena Leaderboard (LMSYS)](https://arena.ai/) --- # Safety and Alignment: A Preview ### The landscape of risks: - **Prompt injection:** adversarial inputs that hijack model behavior - **Jailbreaking:** bypassing safety training to produce harmful outputs - **Data poisoning:** compromised training data leading to backdoors - **Misuse:** generating disinformation, malware, social engineering content - **Autonomy risks:** agents taking unintended actions in the real world ### Current mitigation approaches: - **RLHF / Constitutional AI:** training models to be helpful, harmless, and honest - **System prompts & guardrails:** runtime constraints on model behavior - **Red-teaming:** adversarial testing before deployment - **Content filtering:** input/output classifiers that catch harmful content - **Sandboxing:** limiting what agents can access and execute ### Why reasoning models add new dimensions: - Longer reasoning chains = more surface area for manipulation - Autonomous agents can take irreversible real-world actions - Tool use means models interact with external systems - "Thinking" tokens may contain reasoning the developer cannot inspect (o1) > We will dedicate Week 12 to safety, red-teaming, and responsible deployment practices. ## Sources - [Training Language Models to Follow Instructions with Human Feedback — RLHF (Ouyang et al., 2022)](https://arxiv.org/abs/2203.02155) - [Constitutional AI: Harmlessness from AI Feedback (Bai et al., 2022)](https://arxiv.org/abs/2212.08073) - [OpenAI o1 System Card (OpenAI, 2024)](https://arxiv.org/abs/2412.16720) --- # Course Roadmap: 15 Weeks ## Weeks 1-3: Foundations - Week 1: The 2026 AI Landscape (today) - Week 2: Prompt Engineering & Structured Outputs - Week 3: APIs, SDKs, and Model Integration ## Weeks 4-6: Core Patterns - Week 4: Retrieval-Augmented Generation (RAG) - Week 5: Function Calling & Tool Use - Week 6: Agents and Autonomous Systems ## Weeks 7-9: Building Systems - Week 7: Evaluation and Testing - Week 8: Compound AI Systems & Orchestration - Week 9: Fine-Tuning and Adaptation ## Weeks 10-12: Production & Safety - Week 10: Deployment, Scaling, and Cost Optimization - Week 11: Multimodal Applications - Week 12: Safety, Red-Teaming, and Responsible AI ## Weeks 13-15: Capstone - Week 13: Advanced Topics (inference-time compute, open models) - Week 14: Capstone Project Work - Week 15: Capstone Presentations --- # Key Takeaways ## The field shifted The frontier moved from "bigger models" to "smarter inference." Inference-time compute (o3, R1, extended thinking) is the most important development since the transformer. ## No single best model Model selection is now a multi-dimensional optimization problem: cost, latency, reasoning depth, context length, privacy, and deployment constraints all matter. ## Systems over models The winning approach is compound AI systems -- routing, orchestration, tool use, and verification -- not a single monolithic model call. ## Evaluation is your moat Public benchmarks are saturating and unreliable. Your ability to build domain-specific evaluations is what separates production systems from demos. ## Open-weight models are real DeepSeek R1, Llama 3, Qwen 2.5, and others make self-hosted, private, customizable AI systems practical for the first time. ## The skills gap is changing The bottleneck is no longer API access -- it is the ability to design, evaluate, and deploy systems that use these capabilities effectively. --- # Key Papers and Readings | Paper / Resource | Year | Why It Matters | |-----------------|------|----------------| | [Attention Is All You Need (Vaswani et al.)](https://arxiv.org/abs/1706.03762) | 2017 | The transformer architecture that started it all | | [Scaling Laws for Neural Language Models (Kaplan et al.)](https://arxiv.org/abs/2001.08361) | 2020 | Established training-time scaling laws | | [Chain-of-Thought Prompting (Wei et al.)](https://arxiv.org/abs/2201.11903) | 2022 | Showed reasoning emerges from step-by-step prompting | | [Training Compute-Optimal LLMs (Hoffmann et al., Chinchilla)](https://arxiv.org/abs/2203.15556) | 2022 | Optimal balance of model size and training data | | [GPT-4 Technical Report (OpenAI)](https://arxiv.org/abs/2303.08774) | 2023 | Frontier multimodal model capabilities | | [Scaling LLM Test-Time Compute (Snell et al.)](https://arxiv.org/abs/2408.03314) | 2024 | Formal framework for inference-time scaling | | [DeepSeek-R1 Technical Report](https://arxiv.org/abs/2501.12948) | 2025 | Open-weight reasoning via RL, emergent CoT | | [OpenAI o1 System Card](https://arxiv.org/abs/2412.16720) | 2024 | Safety analysis of reasoning models | | [The Shift from Models to Compound AI Systems (Zaharia et al.)](https://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/) | 2024 | Why systems beat individual models | ### Recommended weekly reading: - Subscribe to: [The Batch (Andrew Ng)](https://www.deeplearning.ai/the-batch/), [Interconnects (Nathan Lambert)](https://www.interconnects.ai/), [Simon Willison's blog](https://simonwillison.net/) - Follow: arXiv cs.CL, cs.AI new submissions --- # Week 1 Complete ## Next week: Prompt Engineering & Structured Outputs ### Before next class: - Read the DeepSeek-R1 technical report (Section 3: RL approach) - Set up API access for at least two providers (OpenAI + one other) - Try the same complex prompt on GPT-4o, o3, Claude 4, and DeepSeek R1 -- compare the outputs **Questions?**