The AI developer stack

Slide 1 of 31Open this slide

The Developer Stack

Architectural Design for AI Applications

Week 3 — Applied AI Course, Spring 2026

"The best framework is the one you don't have to rewrite in six months."

Slide 2 of 31Open this slide

Why Stack Choice Matters

The Cost of Rewrites

• The AI framework landscape changes every 3-6 months — new entrants, breaking changes, abandoned projects
• A bad framework choice can mean weeks of rewrite when requirements shift
• Production AI apps have unique constraints: non-determinism, latency, cost per request

What Goes Wrong

• Over-abstraction: Framework hides the LLM call so deeply you can't debug token usage
• Lock-in: Your prompt logic is tangled with framework-specific state management
• Wrong level of complexity: Using an orchestration engine for a single prompt-response app

The Goal This Week

• Understand the layers of the AI application stack
• Compare LangGraph and PydanticAI in depth — the two strongest contenders for production Python
• Learn architectural patterns so you pick the right one before writing code
• Know what production concerns you'll hit and how to handle them

Slide 3 of 31Open this slide

The AI Application Stack

Every AI application has these layers, whether you make them explicit or not.

Key insight: You can swap layers independently. A good architecture keeps each layer replaceable.

Sources

• LangGraph Documentation
• PydanticAI Documentation
• OpenAI API Documentation
• Anthropic API Documentation
• LiteLLM — Unified LLM API (GitHub)

Slide 4 of 31Open this slide

Decision Map for Choosing Your Stack

Step 1: How many LLM calls per user request?

• One call → Direct API with structured output. No framework needed.
• 2-3 sequential calls → Simple chain. Consider PydanticAI or bare SDK.
• Many calls with branching → You need orchestration. Continue to Step 2.

Step 2: Is the control flow dynamic or static?

• Static (always the same steps) → Pipeline pattern. PydanticAI or simple functions.
• Dynamic (model decides next step) → Agent pattern. Continue to Step 3.

Step 3: Do you need persistent state across turns?

• No (stateless request/response) → PydanticAI agent with tools.
• Yes (multi-turn, checkpointing, human-in-the-loop) → LangGraph with state persistence.

Step 4: How many agents coordinate?

• Single agent → PydanticAI or Claude Agent SDK.
• Multiple agents → LangGraph orchestrator or custom coordination layer.

Rule of thumb: Start with the simplest option. Upgrade when you hit a real limitation, not a hypothetical one.

Slide 5 of 31Open this slide

The Cambrian Explosion of AI Frameworks

The Current Landscape (Spring 2026)

• LangChain / LangGraph — The incumbent. Massive ecosystem, steepest learning curve.
• LlamaIndex Workflows — Once a retrieval library; Workflows turned it into an event-driven, full-stack app framework with first-class state management, workflow orchestration, and structured event passing. Still the strongest data/ingestion layer.
• CrewAI — Multi-agent framework with role-based agents. Good for defined team workflows.
• Microsoft Agent Framework (MAF) — The AutoGen + Semantic Kernel merger (announced late 2025). Targets .NET enterprise: type-safe plugins, middleware, built-in observability, plus AutoGen's multi-agent patterns.
• PydanticAI — Type-safe agents from the Pydantic team. Clean, Pythonic, production-ready.
• Claude Agent SDK — Anthropic's native agent framework. First-class tool use and streaming.
• Instructor — Lightweight structured output. Wraps any LLM client.
• Mirascope — Pythonic LLM toolkit with provider-agnostic design.

MCP Is Reshaping the Stack

• Model Context Protocol (MCP) standardized tool integration, reducing the need for heavy framework abstractions just to call tools
• Agent SDKs like Composio and Prefect now ship 500+ pre-built MCP integrations, so much "framework" value is moving into the protocol + SDK layer

What to Actually Use

Opinion: Most applications need far less framework than developers think.

Sources

• LangChain Documentation
• LangGraph Documentation
• LlamaIndex Documentation
• CrewAI Documentation
• AutoGen — Microsoft (GitHub)
• 10 AI Agent Frameworks You Should Know in 2026 — incl. Microsoft Agent Framework (AtnoForGenAI, 2026)
• LlamaIndex vs LangChain: Which to Choose in 2026 (Contabo)
• LLM Frameworks Replaced by Agent SDKs and MCP (MindStudio, 2026)
• PydanticAI Documentation
• Anthropic Claude SDK Documentation
• Instructor (GitHub)
• Mirascope Documentation

Slide 6 of 31Open this slide

LangChain & LangGraph Ecosystem

LangChain: The Original

• Released late 2022, became the default starting point for LLM applications
• Chains: Sequential composition of prompts, LLM calls, and parsers
• Agents: LLM decides which tools to call via ReAct-style loops
• Criticism: Deep abstraction layers, hard to debug, frequent breaking changes
• 2026 packaging: import langchain-core + langgraph rather than the monolithic langchain package — you pull in only the orchestration primitives you need

Division of Labor with LlamaIndex (2026)

By mid-2026 the production consensus is to use the two together, not pick one:

• LangChain / LangGraph owns orchestration — control flow, agents, stateful workflows
• LlamaIndex Workflows owns data and ingestion — loading, indexing, retrieval
• Most production stacks combine both rather than forcing one framework to do everything

LangGraph: The Evolution

• LangGraph is LangChain's answer to complex, stateful workflows
• Built on a directed graph model — nodes are functions, edges are transitions
• Key features that differentiate it:
  • Typed state that flows through the graph
  • Conditional edges — the LLM or code decides the next node
  • Checkpointing — save and resume graph execution at any point
  • Human-in-the-loop — pause execution, get approval, continue
  • Streaming — stream tokens from any node in the graph

When LangGraph Earns Its Complexity

• Multi-turn conversations with branching logic
• Workflows requiring approval gates or human review
• Long-running tasks that must survive server restarts
• Systems where you need to replay or debug exact execution paths

Sources

• LangChain Documentation
• LangGraph Documentation
• LangSmith Observability Platform
• LangChain vs LlamaIndex: 2026 Update (Zen van Riel)
• ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., 2022)

Slide 7 of 31Open this slide

LangGraph Deep Dive

State Machine Architecture

Core Concepts

from langgraph.graph import StateGraph, START, END
from typing import TypedDict, Literal

class State(TypedDict):
    query: str
    intent: str
    context: list[str]
    response: str

graph = StateGraph(State)
graph.add_node("classify", classify_intent)
graph.add_node("retrieve", retrieve_docs)
graph.add_node("generate", generate_response)

graph.add_edge(START, "classify")
graph.add_conditional_edges("classify", route_by_intent)
graph.add_edge("retrieve", "generate")
graph.add_edge("generate", END)

app = graph.compile(checkpointer=MemorySaver())

Sources

• LangGraph Documentation
• LangGraph StateGraph API Reference

Slide 8 of 31Open this slide

PydanticAI: Type-Safe Agents

Philosophy

• Built by the Pydantic team — the same people behind FastAPI's data validation
• Design principle: Python-first, not framework-first
• Uses standard Python patterns: type hints, dataclasses, dependency injection
• You write normal Python functions; PydanticAI makes them available to the LLM

Core Concepts

• Agent: A configured LLM with a system prompt, tools, and a result type
• Tools: Python functions decorated with @agent.tool — type hints become the schema
• Dependencies: Injected context (database connections, user info) — not hardcoded globals
• Structured Results: Pydantic models as return types — validated automatically
• Model-agnostic: Works with OpenAI, Anthropic, Gemini, Groq, Ollama out of the box

Why PydanticAI Wins for Many Use Cases

• Type safety end-to-end: Your IDE catches errors before runtime
• Testable: Dependency injection means you can mock everything
• Minimal magic: Read the source and understand what happens
• Fast iteration: Change a tool function, re-run — no graph to rewire
• Production-ready: Built-in streaming, retries, and usage tracking

When PydanticAI Is Not Enough

• No built-in graph execution or checkpointing
• Multi-agent coordination requires custom code
• No visual graph editor or execution replay

Sources

• PydanticAI Documentation
• Pydantic Documentation
• FastAPI Documentation

Slide 9 of 31Open this slide

PydanticAI Code Patterns

Basic Agent with Structured Output

from pydantic import BaseModel
from pydantic_ai import Agent

class CityInfo(BaseModel):
    name: str
    country: str
    population: int
    fun_fact: str

agent = Agent(
    "openai:gpt-4o",
    result_type=CityInfo,
    system_prompt="You are a geography expert. Return accurate city data.",
)

result = await agent.run("Tell me about Tokyo")
print(result.data)  # CityInfo(name='Tokyo', country='Japan', ...)

Agent with Tools and Dependencies

from dataclasses import dataclass
from pydantic_ai import Agent, RunContext

@dataclass
class Deps:
    db: DatabasePool
    user_id: str

agent = Agent("anthropic:claude-sonnet-4-20250514", deps_type=Deps)

@agent.tool
async def lookup_order(ctx: RunContext[Deps], order_id: str) -> str:
    """Look up an order by its ID."""
    row = await ctx.deps.db.fetchrow(
        "SELECT * FROM orders WHERE id = $1 AND user_id = $2",
        order_id, ctx.deps.user_id,
    )
    return f"Order {order_id}: status={row['status']}, total=${row['total']}"

@agent.tool
async def cancel_order(ctx: RunContext[Deps], order_id: str) -> str:
    """Cancel an order if it hasn't shipped yet."""
    await ctx.deps.db.execute(
        "UPDATE orders SET status='cancelled' WHERE id=$1", order_id
    )
    return f"Order {order_id} cancelled."

result = await agent.run(
    "Cancel my order #1234",
    deps=Deps(db=pool, user_id="u_abc"),
)

Key Pattern: Dependency Injection Makes Testing Easy

# In tests — swap real DB for a mock
mock_deps = Deps(db=MockDB(), user_id="test_user")
result = await agent.run("Cancel order #1234", deps=mock_deps)

Sources

• PydanticAI Documentation — Agents
• PydanticAI Dependencies & Tool Use
• Pydantic BaseModel Documentation

Slide 10 of 31Open this slide

Anthropic's Claude Agent SDK

What It Is

• Anthropic's official Python SDK for building agents with Claude
• First-class support for tool use, streaming, and multi-turn conversations
• Thin layer over the Messages API — you stay close to the metal

Code Example: Agent with Tools

import anthropic

client = anthropic.Anthropic()

tools = [
    {
        "name": "get_weather",
        "description": "Get current weather for a city",
        "input_schema": {
            "type": "object",
            "properties": {
                "city": {"type": "string", "description": "City name"},
            },
            "required": ["city"],
        },
    }
]

messages = [{"role": "user", "content": "What's the weather in Paris?"}]

# Agent loop: call until no more tool use
while True:
    response = client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=1024,
        tools=tools,
        messages=messages,
    )
    if response.stop_reason == "tool_use":
        tool_block = next(b for b in response.content if b.type == "tool_use")
        result = execute_tool(tool_block.name, tool_block.input)
        messages.append({"role": "assistant", "content": response.content})
        messages.append({
            "role": "user",
            "content": [{"type": "tool_result",
                         "tool_use_id": tool_block.id, "content": result}],
        })
    else:
        break  # Final text response

When to Use the Direct SDK

• You want maximum control over the conversation loop
• You're building Claude-specific features (extended thinking, PDF input, citations)
• You don't need multi-model support
• You prefer explicit code over framework abstractions

Sources

• Anthropic API Documentation
• Anthropic Python SDK — Messages API
• Anthropic Tool Use Documentation

Slide 11 of 31Open this slide

Framework Comparison: Head to Head

LangGraph

• Paradigm: State machine with typed graph
• Strength: Complex, stateful, multi-step workflows
• State management: Built-in checkpointing, persistence
• Learning curve: Steep — graph concepts, state reducers
• Debugging: LangSmith integration, graph visualization
• Best for: Workflows needing human-in-the-loop, replay, branching

PydanticAI

• Paradigm: Type-safe agents with dependency injection
• Strength: Clean single-agent patterns, structured output
• State management: You manage it (just Python)
• Learning curve: Gentle — it's just Python + type hints
• Debugging: Standard Python debugging, Logfire integration
• Best for: Tool-using agents, structured extraction, API backends

Our recommendation: Start with PydanticAI. Move to LangGraph when you need stateful orchestration. Use direct SDK for vendor-specific features.

Sources

• LangGraph Documentation
• PydanticAI Documentation
• LangSmith Platform
• Pydantic Logfire
• OpenAI API Documentation
• Anthropic API Documentation

Slide 12 of 31Open this slide

Pattern 1: Simple Chain

The most common pattern. Most LLM applications are just this.

When to Use

• Single question → single answer
• Text classification, summarization, extraction
• Any task that needs one LLM call

Implementation (Plain Python)

from pydantic_ai import Agent
from pydantic import BaseModel

class Sentiment(BaseModel):
    label: str  # "positive", "negative", "neutral"
    confidence: float
    reasoning: str

agent = Agent(
    "openai:gpt-4o-mini",
    result_type=Sentiment,
    system_prompt="Analyze the sentiment of the given text.",
)

result = await agent.run("This product exceeded my expectations!")
# Sentiment(label='positive', confidence=0.95, reasoning='...')

Cost: ~1 LLM call. Latency: 0.5-2s. Complexity: Minimal.

Do not reach for a framework if this is all you need.

Slide 13 of 31Open this slide

Pattern 2: Router

Classify the input, then route to specialized handlers.

Why Route?

• Use cheap models (GPT-4o-mini) for simple queries, expensive models for complex ones
• Specialized system prompts per category improve quality dramatically
• Route to different tools based on intent — not every query needs every tool

Implementation

from pydantic_ai import Agent
from enum import Enum

class Intent(str, Enum):
    FAQ = "faq"
    TECHNICAL = "technical"
    SALES = "sales"

classifier = Agent("openai:gpt-4o-mini", result_type=Intent,
    system_prompt="Classify the user's intent.")

handlers = {
    Intent.FAQ: faq_agent,
    Intent.TECHNICAL: technical_agent,
    Intent.SALES: sales_agent,
}

intent = (await classifier.run(user_input)).data
response = await handlers[intent].run(user_input)

Cost: 2 LLM calls (classifier is cheap). Latency: 1-3s. Big win: Cost optimization + quality.

Slide 14 of 31Open this slide

Pattern 3: RAG Pipeline

Retrieve relevant context, augment the prompt, then generate.

Key Decisions in RAG

• Chunking strategy: Fixed-size, semantic, document-structure-aware
• Embedding model: OpenAI text-embedding-3-small, Cohere, open-source (BGE, E5)
• Vector store: Postgres+pgvector (simple), Pinecone/Weaviate/Qdrant (scale)
• Retrieval: Top-k similarity, hybrid (BM25 + vector), reranking with Cohere/ColBERT
• Context window: Stuff all chunks vs. map-reduce vs. iterative refinement

Minimal RAG Example

# Retrieval
chunks = await vector_store.similarity_search(query, k=5)
context = "\n---\n".join(c.text for c in chunks)

# Augmented generation
agent = Agent("openai:gpt-4o", system_prompt=f"""
Answer based on the provided context. Cite sources.
Context:
{context}
""")
result = await agent.run(query)

Gotcha: RAG quality depends 80% on retrieval quality, 20% on generation. Invest in chunking and retrieval before prompt engineering.

Sources

• pgvector — Vector Similarity Search for Postgres (GitHub)
• Pinecone Vector Database
• Weaviate Vector Database
• Qdrant Vector Search Engine
• Cohere Rerank API
• ColBERT: Efficient and Effective Passage Search (Khattab & Zaharia, 2020)
• OpenAI Embeddings API

Slide 15 of 31Open this slide

Pattern 4: Agent Loop

The model decides what to do next. Plan, act, observe, repeat.

This Is What "Agents" Actually Are

• The LLM is in the driver's seat — it picks which tool to call and when to stop
• Each iteration adds to the conversation history (growing context)
• The loop runs until: the model returns a final answer, a max iteration limit, or an error

Critical Guardrails

• Max iterations: Always set a cap (5-10 for most use cases)
• Token budget: Track cumulative tokens; abort if budget exceeded
• Tool result size: Truncate large tool outputs before feeding back
• Timeout: Wall-clock timeout for the entire agent run

agent = Agent("anthropic:claude-sonnet-4-20250514",
    system_prompt="You are a research assistant with access to search and calculator.",
    retries=2,
)
# PydanticAI handles the loop internally — tools are called until done
result = await agent.run("What's the GDP per capita of the top 3 EU economies?")

Warning: Agent loops are the most expensive pattern. Each iteration is a full LLM call with growing context. Monitor costs.

Sources

• PydanticAI Documentation — Agents
• ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., 2022)
• Anthropic Building Effective Agents

Slide 16 of 31Open this slide

Pattern 5: Multi-Step Pipeline

A deterministic chain of transformations, each using an LLM.

Key Difference from Agent Loop

• Pipeline: You decide the steps at design time. Steps always run in order.
• Agent loop: The model decides the steps at runtime.

Real Example: Document Processing Pipeline

from pydantic import BaseModel
from pydantic_ai import Agent

class ExtractedData(BaseModel):
    entities: list[str]
    dates: list[str]
    amounts: list[float]

class EnrichedData(BaseModel):
    entities: list[dict]  # With metadata from DB lookup
    timeline: list[dict]
    total_amount: float

# Step 1: Extract structured data
extractor = Agent("openai:gpt-4o-mini", result_type=ExtractedData,
    system_prompt="Extract entities, dates, and monetary amounts.")

# Step 2: Enrich with external data (code, not LLM)
async def enrich(data: ExtractedData) -> EnrichedData:
    entities = [await lookup_entity(e) for e in data.entities]
    return EnrichedData(entities=entities, ...)

# Step 3: Generate summary
summarizer = Agent("openai:gpt-4o", result_type=str,
    system_prompt="Write an executive summary from the enriched data.")

# Pipeline execution
extracted = (await extractor.run(document_text)).data
enriched = await enrich(extracted)
summary = (await summarizer.run(str(enriched))).data

Pro tip: Mix LLM steps with deterministic code steps. Not every transformation needs an LLM.

Slide 17 of 31Open this slide

Pattern 6: Orchestrator

A coordinator that delegates to multiple agents or sub-chains.

LangGraph Implementation

from langgraph.graph import StateGraph, START, END

class OrchestratorState(TypedDict):
    task: str
    plan: list[str]
    research_output: str
    code_output: str
    review_output: str
    final_output: str

graph = StateGraph(OrchestratorState)
graph.add_node("plan", plan_subtasks)
graph.add_node("research", run_research_agent)
graph.add_node("code", run_code_agent)
graph.add_node("review", run_review_agent)
graph.add_node("synthesize", combine_outputs)

graph.add_edge(START, "plan")
# Fan out to parallel agents
graph.add_edge("plan", "research")
graph.add_edge("plan", "code")
graph.add_edge("research", "review")
graph.add_edge("code", "review")
graph.add_edge("review", "synthesize")
graph.add_edge("synthesize", END)

When to Use

• Tasks that naturally decompose into independent sub-tasks
• When different sub-tasks need different tools or models
• When you want parallel execution for speed

Complexity warning: This is the most complex pattern. Each sub-agent is its own agent loop. Debug carefully.

Sources

• LangGraph Documentation
• Anthropic Building Effective Agents

Slide 18 of 31Open this slide

Choosing the Right Pattern

Start Here: What does your application do?

• Single transformation (classify, extract, summarize) → Simple Chain (Pattern 1)
• Different behavior per input type → Router (Pattern 2)
• Answer questions from a knowledge base → RAG (Pattern 3)

If you need multiple LLM calls...

• Steps known at design time → Pipeline (Pattern 5)
• Model decides what to do next → Agent Loop (Pattern 4)
• Multiple specialized agents → Orchestrator (Pattern 6)

The Complexity-Value Tradeoff

The Golden Rule

Use the simplest pattern that solves your problem. Complexity is not a feature.

• 80% of production LLM apps are Pattern 1 or Pattern 2
• Most apps that think they need agents actually need a pipeline
• If you can avoid giving the model control flow decisions, do it

Slide 19 of 31Open this slide

Observability for LLM Applications

Why LLM Observability Is Different

• Traditional apps: request in, response out, log the status code
• LLM apps: you need to see the prompt, completion, token count, latency, and cost for every call
• Agent loops make N calls per request — you need the full trace

The Observability Stack

• LangSmith — LangChain's tracing platform. Works with LangGraph natively. Can instrument any LLM call.
• Langfuse — Open-source LLM observability. Self-hostable. Model-agnostic.
• Arize Phoenix — Open-source traces and evals. Strong on embeddings visualization.
• Logfire — From the Pydantic team. Native PydanticAI integration. Built on OpenTelemetry.
• Braintrust — Logging, evals, and prompt playground in one.

What to Trace

• Every LLM call: model, prompt, completion, tokens (input/output), latency, cost
• Tool calls: which tool, input arguments, output, duration
• Agent iterations: how many loops, which tools called per loop
• End-to-end request: total latency, total cost, total tokens

Minimal Setup with Logfire

import logfire
logfire.configure()
logfire.instrument_pydantic_ai()

# Now every agent.run() is automatically traced
result = await agent.run("What's the status of order #1234?")

Non-negotiable: Do not ship an LLM application without tracing. You will not be able to debug production issues without it.

Sources

• LangSmith Observability Platform
• Langfuse — Open-Source LLM Observability
• Arize Phoenix — AI Observability & Evaluation (GitHub)
• Pydantic Logfire
• Braintrust — AI Observability Platform

Slide 20 of 31Open this slide

Error Handling for LLM APIs

LLM APIs Fail in Ways Traditional APIs Don't

• Rate limits (429): You'll hit them. Every provider has them.
• Timeout: Large prompts or complex reasoning can take 30-60+ seconds.
• Malformed output: The model returns JSON that doesn't match your schema.
• Content filtering: Model refuses the request due to safety filters.
• Context length exceeded: Your prompt + history is too long.
• Service outages: Even OpenAI and Anthropic have downtime.

Defense Layers

1. Retries with Exponential Backoff

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential(min=1, max=30))
async def call_llm(prompt: str) -> str:
    return await agent.run(prompt)

2. Fallback Models

async def call_with_fallback(prompt: str) -> str:
    try:
        return await primary_agent.run(prompt)  # Claude Sonnet
    except Exception:
        return await fallback_agent.run(prompt)  # GPT-4o-mini

3. Circuit Breaker

• After N consecutive failures, stop calling the API for a cooldown period
• Prevents cascading failures and runaway costs
• Libraries: circuitbreaker, aiobreaker, or custom implementation

4. Structured Output Validation

# PydanticAI handles this — if output doesn't validate,
# it automatically re-prompts the model with the validation error
agent = Agent("openai:gpt-4o", result_type=MyModel, retries=2)

The Error Handling Checklist

• Retries with backoff for transient failures
• Fallback model for provider outages
• Timeout on every LLM call (30s default, adjust per use case)
• Input validation before calling the LLM (check token count)
• Output validation after (Pydantic models, not string parsing)
• Graceful degradation in the UI when all else fails

Sources

• Tenacity — Retrying Library for Python (GitHub)
• PydanticAI Documentation — Retries & Validation
• OpenAI API Error Handling
• Anthropic API Error Handling

Slide 21 of 31Open this slide

Streaming Responses

Why Streaming Matters

• LLM responses take 2-30 seconds to fully generate
• Without streaming, users stare at a blank screen — perceived latency is terrible
• With streaming, the first token arrives in ~200ms — users start reading immediately
• Streaming reduces time-to-first-byte (TTFB) from seconds to milliseconds

Implementation Approaches

Server-Sent Events (SSE) — The Standard

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from pydantic_ai import Agent

app = FastAPI()
agent = Agent("openai:gpt-4o")

@app.get("/chat")
async def chat(query: str):
    async def event_stream():
        async with agent.run_stream(query) as result:
            async for text in result.stream_text(delta=True):
                yield f"data: {json.dumps({'text': text})}\n\n"
        yield "data: [DONE]\n\n"
    return StreamingResponse(event_stream(), media_type="text/event-stream")

WebSockets — For Bidirectional Communication

• Better for multi-turn chat where client sends messages during generation
• More complex to implement, but lower overhead for ongoing conversations

Streaming with Structured Output

• You cannot stream structured (JSON) output token-by-token meaningfully
• Common pattern: stream a text explanation, then return structured data at the end
• PydanticAI supports partial streaming — yields validated partial results as fields complete

UX Considerations

• Show a typing indicator before the first token arrives
• Buffer a few tokens before rendering to avoid single-character flicker
• Provide a "Stop generating" button for long responses
• Display token/cost counters in development mode

Sources

• PydanticAI Documentation — Streaming
• FastAPI StreamingResponse Documentation
• MDN Web Docs — Server-Sent Events

Slide 22 of 31Open this slide

Caching Strategies

Caching is the single biggest lever for reducing LLM costs and latency.

Exact Match Cache

The simplest approach. Hash the prompt, cache the response.

• When: Identical prompts repeat often (classification, extraction on templates)
• Implementation: Redis or in-memory dict keyed by hash(model + prompt)
• Hit rate: High for structured/templatized inputs, low for free-form chat
• TTL: Hours to days depending on freshness requirements

Semantic Cache

Embed the prompt, find similar cached prompts within a threshold.

• When: Users ask similar questions in different words
• Implementation: Vector store of prompt embeddings, cosine similarity threshold (0.95+)
• Risk: Returning a cached answer to a subtly different question — tune threshold carefully
• Tools: GPTCache, Redis with vector search, custom with pgvector

Prompt Caching (Provider-Level)

Anthropic and OpenAI cache long prompt prefixes server-side.

• Anthropic: Automatic for prompts sharing a prefix. Up to 90% cost reduction on cached tokens.
• OpenAI: Automatic prefix caching on prompts > 1024 tokens. 50% discount on cached input tokens.
• When: System prompts, large context documents, or RAG contexts that repeat across calls
• Action: Structure prompts so the static part (system prompt, context) comes first

KV Cache (Self-Hosted)

For self-hosted models, the attention KV cache avoids recomputing previous tokens.

• When: Running Llama, Mistral, etc. on your own GPUs
• Implementation: vLLM, TGI handle this automatically with PagedAttention
• Impact: 2-4x throughput improvement
• Relevance: Only matters if you self-host; API providers handle this for you

Sources

• Anthropic Prompt Caching Documentation
• OpenAI Prompt Caching Guide
• GPTCache — Semantic Cache for LLMs (GitHub)
• vLLM — High-Throughput LLM Serving (GitHub)
• PagedAttention: Efficient Memory Management for LLM Serving (Kwon et al., 2023)

Slide 23 of 31Open this slide

Cost Management

The Cost Equation

Cost = (input_tokens * input_price) + (output_tokens * output_price)

Prices per million tokens (as of early 2026):

Cost Optimization Strategies

1. Model Routing

• Route simple queries to cheap models (GPT-4o-mini, Haiku, Gemini Flash)
• Reserve expensive models for complex tasks
• A router paying $0.15/M tokens to save $10/M tokens on 60% of requests is a huge win

2. Prompt Optimization

• Shorter prompts = fewer input tokens = lower cost
• Remove verbose system prompts — be concise
• Use structured output to minimize output tokens (JSON, not prose)

3. Token Budgets

# Track usage across agent runs
result = await agent.run("Analyze this document")
print(f"Tokens used: {result.usage().total_tokens}")
print(f"Cost: ${result.usage().total_tokens * price_per_token:.4f}")

4. Budget Controls

• Set per-request, per-user, and per-day spending limits
• Alert when spend exceeds thresholds
• Kill switch: disable expensive features when budget is exhausted

Real-World Benchmark

A support chatbot handling 10,000 conversations/day:

• Without optimization: ~$500/day (GPT-4o for everything)
• With routing + caching: ~$80/day (70% Haiku, 25% Sonnet, 5% GPT-4o, 40% cache hit)

Sources

• OpenAI API Pricing
• Anthropic Claude API Pricing
• Google Gemini API Pricing

Slide 24 of 31Open this slide

Testing AI Applications

The Testing Pyramid for LLM Apps

Traditional testing still applies, but you need an additional layer: evaluations.

1. Unit Tests — Test Your Code, Not the Model

# Test tool functions independently
async def test_lookup_order():
    mock_db = MockDB(orders={"1234": {"status": "shipped", "total": 59.99}})
    deps = Deps(db=mock_db, user_id="test")
    result = await lookup_order(RunContext(deps=deps), "1234")
    assert "shipped" in result
    assert "59.99" in result

2. Integration Tests — Test with a Real (or Mocked) Model

# Use a deterministic model for integration tests
from pydantic_ai.models.test import TestModel

agent = Agent("openai:gpt-4o", result_type=Sentiment)

# Override with test model that returns predictable output
with agent.override(model=TestModel()):
    result = await agent.run("Great product!")
    assert isinstance(result.data, Sentiment)

3. Eval Suites — Test Model Quality

• What to eval: Accuracy, relevance, safety, format compliance, latency
• Golden dataset: Curated input-output pairs with human-verified expected results
• LLM-as-judge: Use a stronger model to grade outputs of a weaker model
• Run evals on every prompt change, model upgrade, or framework update

Eval Frameworks

The Eval Workflow

1. Build a golden dataset (50-200 examples covering edge cases)
2. Define metrics (exact match, LLM-as-judge, custom scoring)
3. Run evals on every change to prompts, models, or tools
4. Track scores over time — regressions are the enemy
5. Add failing production examples to your golden dataset

Non-negotiable: If you don't have evals, you're doing vibe-based development. That doesn't scale.

Sources

• PydanticAI TestModel Documentation
• Promptfoo — LLM Eval Framework
• Braintrust — Evals & Observability Platform
• DeepEval — LLM Evaluation Framework (GitHub)

Slide 25 of 31Open this slide

Deployment Patterns

Serverless (AWS Lambda, Cloud Functions, Vercel)

• Pros: Zero ops, auto-scaling, pay per invocation
• Cons: Cold starts (1-3s), 15-min timeout limits, no persistent connections
• Best for: Simple chain patterns, low-traffic apps, webhook handlers
• Watch out: LLM calls can take 30s+ — Lambda's timeout can bite you
• Stack: API Gateway + Lambda + DynamoDB for state

Containers (Docker + ECS/Cloud Run/K8s)

• Pros: Full control, long-running connections, persistent state
• Cons: You manage scaling, more ops overhead
• Best for: Agent loops, streaming responses, multi-turn chat
• Stack: FastAPI in Docker, deployed to Cloud Run or ECS
• Recommended for most production AI apps

Edge Deployment (Cloudflare Workers, Vercel Edge)

• Pros: Low latency globally, cheap at scale
• Cons: Limited runtime (no Python natively), small memory limits
• Best for: Routing layer, caching proxy in front of LLM APIs
• Pattern: Edge function for auth/cache/routing, origin server for LLM calls

Self-Hosted Models (vLLM, TGI, Ollama)

• Pros: No per-token cost, data stays on your infra, full control
• Cons: GPU costs ($2-8/hr per A100), ops complexity, model updates are manual
• Best for: High-volume apps where API costs exceed GPU costs, strict data residency
• Break-even: Roughly 10M+ tokens/day before self-hosting is cheaper
• Stack: vLLM on NVIDIA GPUs, behind a load balancer

Starting recommendation: FastAPI in a Docker container on Cloud Run. Add complexity only when you outgrow it.

Sources

• FastAPI Documentation
• vLLM — High-Throughput LLM Serving (GitHub)
• Hugging Face Text Generation Inference (GitHub)
• Ollama
• Google Cloud Run Documentation
• Vercel AI SDK

Slide 26 of 31Open this slide

Anti-Patterns

Things that seem smart but will cost you later.

1. Over-Abstraction

• Wrapping every LLM call in 5 layers of abstraction "for flexibility"
• You can't debug what you can't read. Keep the LLM call visible.
• Fix: If you can't explain what your code does in one sentence, simplify it.

2. Framework Lock-In

• Putting business logic inside framework-specific constructs (LangChain's Chain classes, custom graph nodes)
• When the framework changes (and it will), your logic is trapped
• Fix: Business logic in plain Python functions. Framework just wires them together.

3. Premature Agent-ification

• Using an autonomous agent loop when a simple pipeline would work
• Agents are expensive, slow, and unpredictable by design
• Fix: Can you write the steps as a for-loop? Then don't use an agent.

4. Ignoring Cost Until the Bill Arrives

• Running GPT-4o on every request during development, then panicking at the invoice
• Fix: Instrument token counting from day one. Set budget alerts.

5. The "We'll Add Evals Later" Trap

• Shipping without evals means you can't measure the impact of any change
• Fix: Build 20 golden examples before writing any prompt. Run evals in CI.

6. Stuffing Everything Into the System Prompt

• 5,000-token system prompts that try to handle every edge case
• More instructions ≠ better behavior. Models get confused by contradictions.
• Fix: Shorter, focused prompts. Use routing to select the right prompt per task.

Slide 27 of 31Open this slide

The Minimal Viable Stack

Start here. Add complexity only when you have evidence you need it.

Level 0: Prototype (Day 1)

• Model: OpenAI or Anthropic API directly
• Framework: None. Just openai or anthropic SDK.
• Observability: print() statements
• Deployment: Run locally or in a notebook
• Testing: Manual testing in the REPL

Level 1: First Production App

• Model: GPT-4o-mini or Claude Haiku for cost efficiency
• Framework: PydanticAI for structured output and tool use
• Observability: Logfire or Langfuse for tracing
• Deployment: FastAPI in Docker on Cloud Run
• Testing: Unit tests for tools, 50-example eval suite

Level 2: Scaling Up

• Model: Router across multiple models (cheap for simple, expensive for complex)
• Framework: PydanticAI for agents, LangGraph only if you need stateful workflows
• Observability: Full tracing with cost tracking and alerting
• Deployment: Kubernetes or ECS with auto-scaling
• Testing: Comprehensive evals in CI, LLM-as-judge for quality

Level 3: Enterprise

• Model: Self-hosted for high-volume, API for burst capacity
• Framework: Custom orchestration layer on top of PydanticAI/LangGraph
• Observability: Integrated with existing APM (Datadog, New Relic)
• Deployment: Multi-region, failover across providers
• Testing: A/B testing, continuous eval, human review loops

Most teams should live at Level 1 for longer than they think.

Sources

• PydanticAI Documentation
• LangGraph Documentation
• Pydantic Logfire
• Langfuse — Open-Source LLM Observability
• FastAPI Documentation
• OpenAI API Documentation
• Anthropic API Documentation

Slide 28 of 31Open this slide

The Post-Training Stack: Hugging Face TRL v1.0

Most of this week is about inference-time orchestration. When you need to adapt a model's weights, the stack converges on one library.

Hugging Face TRL v1.0 (April 2026)

• The unified post-training library — one API for the whole alignment pipeline (SFT, reward modeling, DPO, GRPO)
• Integrates Unsloth kernels for ~2x training speed and ~70% memory reduction, putting fine-tuning within reach of a single GPU
• The techniques themselves — LoRA/QLoRA, DPO, GRPO, distillation — are covered in depth in Week 13 (Distillation & Fine-Tuning). Here it's just the stack-level answer to "which library."

Where It Fits in the Stack

• Sits below the orchestration and framework layers — it produces the model artifact those layers serve
• Use it when prompting, RAG, and tool use have plateaued and you have labeled or preference data for your domain
• For most teams this is a Level 3 / enterprise concern, not a day-one decision

Fine-tune last, not first: exhaust prompting, retrieval, and tool use before you touch weights.

Sources

• Hugging Face Releases TRL v1.0: Unified Post-Training Stack (MarkTechPost, Apr 2026)

Slide 29 of 31Open this slide

Key Takeaways

Simplicity Wins

Most LLM applications need a simple chain or a router, not an agent framework. Start with the simplest pattern and upgrade only when you hit a real wall. Complexity is a cost, not a feature.

PydanticAI for Agents, LangGraph for Workflows

PydanticAI is the best default for tool-using agents: type-safe, testable, Pythonic. Reach for LangGraph when you need stateful, checkpointed, multi-step workflows with human-in-the-loop. Don't use both where one will do.

Separate Business Logic from Framework

Write your core logic as plain Python functions. The framework should wire things together, not own your code. This protects you when frameworks change, break, or get abandoned.

Observability and Evals Are Non-Negotiable

Instrument tracing from day one. Build an eval suite before writing prompts. If you can't measure it, you can't improve it, and you definitely can't debug it in production.

Cost Is a First-Class Concern

Token costs compound fast. Route to cheap models, cache aggressively, set budget controls, and track spend per feature. The difference between a naive and optimized stack can be 5-10x in cost.

Ship, Then Optimize

Get to production with a minimal stack. Real traffic reveals which optimizations matter. Premature optimization of AI systems is especially wasteful because the model landscape changes quarterly.

Slide 30 of 31Open this slide

Key Resources and Documentation

Framework Documentation

Observability and Evaluation

Architecture References

This Week's Recommended Reading

1. PydanticAI documentation — read the "Getting Started" and "Agents" sections
2. LangGraph "Quick Start" tutorial — build a simple graph to understand the model
3. Anthropic's "Building Effective Agents" blog post — best overview of agent patterns

Slide 31 of 31Open this slide

Next Week

Week 4: Knowledge Representation

Embeddings, vector search, hybrid retrieval, and knowledge graphs for grounding AI systems.

Assignment

Build a tool-using agent with PydanticAI that:

1. Accepts a natural language query
2. Uses at least two tools (e.g., database lookup + web search)
3. Returns a structured Pydantic model as output
4. Includes at least 10 eval cases with a scoring function

Due: Before Week 4 lecture.

Start simple. Add complexity when the evals tell you to.