Small models and model routing

Slide 1 of 30Open this slide

The Small Model Strategy

Right-Sizing AI for Production

Week 12 — Applied AI Course, Spring 2026

Not every nail needs a frontier-model hammer. Learn when a 3B parameter model outperforms — and out-earns — a trillion-parameter one.

Slide 2 of 30Open this slide

The Cost Reality

Frontier Model Pricing (per 1M tokens, as of early 2026)

• GPT-4.1: $2.00 input / $8.00 output
• Claude Sonnet 4: $3.00 input / $15.00 output
• Claude Opus 4: $15.00 input / $75.00 output
• Gemini 2.5 Pro: $1.25–$10.00 input / $2.50–$30.00 output
• o3-pro: $20.00 input / $80.00 output

Small Model Pricing

• GPT-4.1 mini: $0.40 input / $1.60 output
• GPT-4.1 nano: $0.10 input / $0.40 output
• Claude Haiku 3.5: $0.80 input / $4.00 output
• Gemini 2.0 Flash: $0.10 input / $0.40 output
• Self-hosted Llama 3.2 3B: ~$0.02–0.05/M tokens (compute only)

The Math

A pipeline processing 100M tokens/month:

• With Claude Opus 4: ~$4,500/month
• With GPT-4.1 nano: ~$25/month
• With self-hosted Llama 3.2 3B: ~$3/month

That is a 180x–1,500x cost difference for tasks where small models perform comparably.

Slide 3 of 30Open this slide

When Small Models Win

Cost Efficiency

• High-volume, low-complexity tasks (classification, extraction, reformatting)
• Batch processing millions of documents where per-token cost dominates
• Startups and projects with tight budgets

Latency

• Small models respond in 10–50ms vs 500ms–5s for frontier models
• Real-time applications: autocomplete, inline suggestions, chat
• Time-to-first-token matters for user experience

Privacy and Compliance

• On-premise deployment keeps data off third-party servers
• HIPAA, GDPR, SOC2 compliance without data processing agreements
• Air-gapped environments in defense, finance, healthcare

Edge and On-Device

• Mobile apps, IoT devices, embedded systems
• Offline capability — no network dependency
• Browser-based inference with WebGPU/WebLLM

Specialized Tasks

• Fine-tuned small models often beat general-purpose large models on narrow tasks
• Structured output (JSON, SQL, code) with constrained generation
• Domain-specific extraction where the task is well-defined

Slide 4 of 30Open this slide

The Model Size Spectrum

 Parameters    Examples                    Typical Use Cases              Hardware
 ──────────    ────────                    ─────────────────              ────────

   1B          Llama 3.2 1B               Simple classification,         Phone, RPi
               Qwen 2.5 0.5B             text cleanup, routing          1–2 GB RAM
                    │
                    ▼
   3B          Llama 3.2 3B               Extraction, summarization,     Laptop CPU
               Phi-4 mini                 basic Q&A, translation         4 GB RAM
                    │
                    ▼
   7–8B        Llama 3.1 8B               General assistant, code gen,   Single GPU
               Qwen 2.5 7B               RAG, tool use                  8–16 GB VRAM
               Gemma 3 4B/12B
                    │
                    ▼
  13–14B       Qwen 2.5 14B              Complex reasoning, long-form   Single GPU
               Phi-4 14B                 generation, multi-step tasks   16–24 GB VRAM
                    │
                    ▼
  30–70B       Llama 3.3 70B             Near-frontier quality,         Multi-GPU
               Qwen 2.5 72B             complex analysis               40–80 GB VRAM
                    │
                    ▼
 Frontier      GPT-4.1, Claude Opus 4,   Hardest reasoning, creative    API only /
               Gemini 2.5 Pro            writing, novel problems        Data center

Key insight: The right model is the smallest one that meets your quality threshold.

Sources

• Introducing Meta Llama 3 (Meta, 2024)
• Qwen2.5: A Party of Foundation Models (Alibaba, 2024)
• Introducing Phi-3: Redefining What's Possible with SLMs (Microsoft, 2024)
• Phi-4 Technical Report — arXiv:2412.08905 (Microsoft, 2024)
• Gemma 3 (Google, 2025)

Slide 5 of 30Open this slide

Meta Llama 3.2 and Llama 4

Llama 3.2 (Sept 2024) — The Open-Weight Foundation

• 1B and 3B text models — first truly capable sub-4B models from Meta
• 11B and 90B vision-language multimodal variants
• 128K context window across all sizes
• Trained on 9T tokens — massive data investment even for small models
• Optimized for on-device: Qualcomm, MediaTek partnerships at launch

Llama 3.3 70B (Dec 2024)

• Matches Llama 3.1 405B performance at a fraction of the compute
• Demonstrates that distillation from larger models is highly effective

Llama 4 (April 2025)

• Scout: 17B active params, 16 experts (109B total), 10M context window. Meta's release describes a single H100 with Int4 quantization. The 17B active count is not the full weight-memory requirement.
• Maverick: 17B active params, 128 experts (400B total)
• Mixture-of-Experts architecture — only activates a subset per token
• Early-fusion native multimodality (text, image, video)

Why It Matters for Small Model Strategy

• Llama models use version-specific Community Licenses and acceptable-use terms. Review the actual model license for the intended use; do not treat it as Apache 2.0.
• Massive ecosystem: fine-tuning tools, quantized variants, deployment infra
• The 1B and 3B models run comfortably on a smartphone

Sources

• Llama 3.2: Revolutionizing Edge AI and Vision (Meta, 2024)
• The Llama 3 Herd of Models — arXiv:2407.21783 (Meta, 2024)
• The Llama 4 Herd: Natively Multimodal AI Innovation (Meta, 2025)
• Llama model family (Meta, Hugging Face)
• Llama 4 Community License

Slide 6 of 30Open this slide

Qwen 2.5 and Qwen 3

Qwen 2.5 (Sept 2024) — The Quiet Powerhouse

• Full size range: 0.5B, 1.5B, 3B, 7B, 14B, 32B, 72B
• Best-in-class multilingual support (29+ languages)
• Exceptionally strong on math (GSM8K, MATH) and coding (HumanEval)
• Qwen 2.5 7B rivals Llama 3.1 70B on several benchmarks
• Apache 2.0 license for models up to 72B

Qwen 2.5 Coder

• Dedicated coding variants at 1.5B, 7B, 14B, 32B
• Qwen 2.5 Coder 32B matches GPT-4o on code generation tasks
• Supports 90+ programming languages

Qwen 3 / 3.6 (2025–2026)

• Hybrid thinking models: can toggle between "thinking" and "non-thinking" modes
• Sizes from 0.6B to 235B (MoE with 22B active)
• Qwen 3 8B matches Qwen 2.5 72B on reasoning benchmarks
• Qwen 3.6 27B (May 2026) hits 77.2% on SWE-bench, a new high-water mark for an open small model
• 128K context with YaRN-based extension to 1M tokens
• Apache 2.0 license across all sizes

Why Qwen Matters

• Strongest small models for non-English languages
• Math and code performance punches well above weight class
• Fully open weights — no usage restrictions

Sources

• Qwen2.5: A Party of Foundation Models (Alibaba, 2024)
• Qwen2.5 Technical Report — arXiv:2412.15115 (Qwen Team, 2024)
• Qwen2.5-Coder Technical Report — arXiv:2409.12186 (Qwen Team, 2024)
• Qwen3: Think Deeper, Act Faster (Alibaba, 2025)
• Qwen3 Technical Report — arXiv:2505.09388 (Qwen Team, 2025)
• Qwen3 (QwenLM, GitHub)

Slide 7 of 30Open this slide

Microsoft Phi-3 and Phi-4

The "Textbook Quality" Thesis

• Core insight: data quality matters more than data quantity at small scale
• Phi models trained on heavily curated, synthetic, and textbook-style data
• Result: dramatically better performance per parameter than expected

Phi-3 (April 2024)

• Phi-3 mini (3.8B): rivals Mixtral 8x7B and GPT-3.5 on many benchmarks
• Phi-3 small (7B) and Phi-3 medium (14B): competitive with much larger models
• 128K context window support
• Optimized for ONNX Runtime — fast inference on CPU and mobile

Phi-4 (Dec 2024 – Feb 2025)

• Phi-4 (14B): outperforms GPT-4o on MATH and GPQA benchmarks
• Phi-4 mini (3.8B): a benchmark-beating edge model — runs at 15–20 tok/s on an M1 MacBook Air at Q4_K_M, matches GPT-4o on structured-extraction benchmarks, and fits on entry-level hardware
• Heavy use of synthetic data generated by larger models
• Particularly strong at STEM reasoning and structured tasks

Trade-offs

• Narrower training data means weaker on open-ended creative tasks
• Less robust on adversarial/safety benchmarks compared to Llama/Qwen
• Microsoft Research license — check terms for commercial use
• Best suited for STEM, structured extraction, and reasoning pipelines

Sources

• Introducing Phi-3: Redefining What's Possible with SLMs (Microsoft, 2024)
• Phi-3 Technical Report — arXiv:2404.14219 (Microsoft, 2024)
• Introducing Phi-4: Microsoft's Newest Small Language Model (Microsoft, 2024)
• Phi-4 Technical Report — arXiv:2412.08905 (Microsoft, 2024)
• Small Language Models: Phi-4 & Gemma On-Device 2026 (MasterPrompting)

Slide 8 of 30Open this slide

Mistral and Mixtral

Mistral 7B (Sept 2023) — The Model That Started the SLM Revolution

• Outperformed Llama 2 13B on every benchmark at half the size
• Introduced Sliding Window Attention (SWA) for efficient long contexts
• Grouped Query Attention (GQA) for faster inference
• Apache 2.0 license — fully permissive

Mixtral 8x7B (Dec 2023) — Mixture of Experts Goes Mainstream

• 8 expert networks, 2 active per token — 46.7B total, ~13B active
• Matches or beats Llama 2 70B while using 6x less compute per token
• Proved MoE is practical for production deployment

Mistral Small / Nemo (2024–2026)

• Mistral Nemo 12B: collaboration with NVIDIA, strong general-purpose model
• Mistral Small 3.1 24B: optimized for fast API-served inference
• Mistral Small 4 (March 16, 2026): a hybrid model that unifies instruct, reasoning, multimodal, and coding capabilities in a single checkpoint
• Function calling and JSON mode built in
• Tekken tokenizer: more efficient for multilingual and code

MoE Architecture — Why It Matters

• Total parameters determine knowledge capacity (what the model knows)
• Active parameters determine inference cost (what you pay per token)
• MoE gives you big-model knowledge at small-model cost
• Trade-off: higher memory footprint (all experts must be loaded)

Sources

• Mistral 7B (Mistral AI, 2023)
• Mistral 7B — arXiv:2310.06825 (Jiang et al., 2023)
• Mixtral of Experts (Mistral AI, 2023)
• Mixtral of Experts — arXiv:2401.04088 (Jiang et al., 2024)
• Mistral NeMo (Mistral AI & NVIDIA, 2024)
• Mistral Small 3.1 (Mistral AI, 2025)
• Mistral Small (latest) Model Stats (LLM-Stats)

Slide 9 of 30Open this slide

Google Gemma 2 and Gemma 3

Gemma 2 (June 2024)

• Sizes: 2B, 9B, 27B
• Knowledge distillation from larger Gemini models
• 9B model rivals Llama 3 8B and Mistral 7B across benchmarks
• Optimized for TPU, GPU, and CPU inference

Gemma 3 (March 2025) — Built for On-Device

• Sizes: 1B, 4B, 12B, 27B
• Native multimodal: text + image + short video understanding
• 128K context window (up from 8K in Gemma 2)
• ShieldGemma built-in safety classifier
• Runs on a single NVIDIA H100, RTX 3090, or even TPU v5e

Key Differentiators

• Tight integration with Google ecosystem: Vertex AI, Colab, Android
• Gemma 3 4B with quantization runs on Pixel phones
• Strong structured output and function calling support
• "Gemma terms of use" license — permissive for most commercial use

Performance Highlights

• Gemma 3 27B competes with Llama 3.3 70B on reasoning tasks
• Gemma 3 12B beats Qwen 2.5 14B on multimodal understanding
• Designed from the ground up for efficient on-device inference

Sources

• Gemma 2: Improving Open Language Models at a Practical Size — arXiv:2408.00118 (Google DeepMind, 2024)
• Google Launches Gemma 2 (Google, 2024)
• Gemma 3: Google's New Open Model Based on Gemini 2.0 (Google, 2025)
• Gemma 3 Technical Report (Google DeepMind, 2025)

Slide 10 of 30Open this slide

Head-to-Head: Small Model Benchmarks

~3B Parameter Class

~7–14B Parameter Class

Scores are approximate composites from published benchmarks and independent evaluations. Always validate on your own data.

Key Takeaway

The gap between 7B models and frontier models has narrowed dramatically. On focused tasks, the difference can be negligible.

Sources

• MMLU: Measuring Massive Multitask Language Understanding — arXiv:2009.03300 (Hendrycks et al., 2020)
• HumanEval: Evaluating Large Language Models Trained on Code — arXiv:2107.03374 (Chen et al., 2021)
• GSM8K: Training Verifiers to Solve Math Word Problems — arXiv:2110.14168 (Cobbe et al., 2021)
• GPQA: A Graduate-Level Google-Proof Q&A Benchmark — arXiv:2311.12022 (Rein et al., 2023)

Slide 11 of 30Open this slide

When Small Beats Big

The Counterintuitive Results

Fine-tuned small models regularly outperform general-purpose frontier models on specific tasks. This is not a fluke — it is a repeatable pattern.

Documented Examples

• Medical NER: Fine-tuned Llama 3.1 8B outperformed GPT-4 on clinical entity extraction by 8% F1 (Microsoft Research, 2024)
• Legal clause classification: Fine-tuned Phi-3 mini beat GPT-4o by 12% accuracy on contract review tasks
• SQL generation: Fine-tuned Qwen 2.5 Coder 7B matched GPT-4.1 on Spider benchmark while costing 100x less
• Sentiment analysis: A fine-tuned 3B model achieves 95%+ accuracy — no frontier model needed
• Structured extraction: Constrained decoding with a 7B model produces valid JSON 99.9% of the time

Why Does This Happen?

1. Task narrowness: Small models have enough capacity for focused tasks
2. Fine-tuning efficiency: Adapting all parameters of a small model beats prompting a large one
3. Output constraints: Structured tasks reduce the generation space — smaller models excel
4. Data quality: Domain-specific fine-tuning data teaches exactly what is needed

The Rule of Thumb

If your task can be described in a 1-page annotation guide and you have 1,000+ labeled examples, a fine-tuned 7B model will likely match frontier performance.

Sources

• The Llama 3 Herd of Models — arXiv:2407.21783 (Meta, 2024)
• Phi-3 Technical Report — arXiv:2404.14219 (Microsoft, 2024)
• Qwen2.5-Coder Technical Report — arXiv:2409.12186 (Qwen Team, 2024)

Slide 12 of 30Open this slide

Task-to-Model Matching

Classification & Routing

Model size: 1B–3B

• Sentiment analysis, spam detection, intent classification
• Topic categorization, language identification
• Binary/multi-class decisions with structured output
• Fine-tuned models achieve 95%+ accuracy
• Example: Llama 3.2 1B for support ticket routing

Extraction & Transformation

Model size: 3B–7B

• Named entity recognition, PII detection
• JSON/structured data extraction from text
• Text reformatting, normalization, translation
• Constrained generation ensures output validity
• Example: Qwen 2.5 3B for invoice field extraction

Generation & Summarization

Model size: 7B–14B

• Document summarization, email drafting
• Code generation for well-defined tasks
• RAG-based Q&A with grounded responses
• Quality degrades gracefully with model size
• Example: Phi-4 14B for technical documentation Q&A

Complex Reasoning

Model size: 30B+ or Frontier

• Multi-step logical reasoning, planning
• Novel creative writing, open-ended analysis
• Ambiguous tasks requiring world knowledge
• Tasks where errors are expensive
• Example: Claude Opus 4 for legal brief analysis

Slide 13 of 30Open this slide

Benchmarking for YOUR Use Case

Public Leaderboards Are Not Enough

• MMLU, HumanEval, GSM8K measure general capability
• Your production task is specific — leaderboard rankings often do not correlate
• Models optimize for benchmarks (contamination, overfitting)
• Your data distribution, prompt format, and quality bar are unique

Build Custom Evals

Step 1: Define Your Quality Metric

• Accuracy, F1, BLEU, ROUGE — pick what maps to business value
• Include human preference ratings for subjective tasks
• Define a minimum quality threshold (e.g., "95% extraction accuracy")

Step 2: Create a Golden Test Set

• 200–500 representative examples from your actual data
• Cover edge cases, hard examples, and common patterns
• Version control your test set — it is a critical asset

Step 3: Run Structured Comparisons

for model in [gpt-4.1, gpt-4.1-nano, llama-3.2-3b, qwen3-8b]:
    results = evaluate(model, test_set, prompt_template)
    log(model, accuracy, latency_p50, latency_p99, cost_per_1k)

Step 4: Plot the Pareto Frontier

• X-axis: cost per 1,000 requests
• Y-axis: quality score
• Choose the model on the Pareto frontier that meets your quality bar

Tools for Custom Evals

• OpenAI Evals / Inspect AI: framework for structured evaluation
• LangSmith / Braintrust: tracing + eval platforms
• promptfoo: open-source CLI for prompt and model comparison
• Weights & Biases: experiment tracking for model selection

Sources

• MMLU: Measuring Massive Multitask Language Understanding — arXiv:2009.03300 (Hendrycks et al., 2020)
• HumanEval: Evaluating Large Language Models Trained on Code — arXiv:2107.03374 (Chen et al., 2021)
• GSM8K: Training Verifiers to Solve Math Word Problems — arXiv:2110.14168 (Cobbe et al., 2021)
• Inspect AI — LLM Evaluation Framework (UK AISI)
• LangSmith — Tracing and Eval Platform (LangChain)
• Braintrust — AI Observability and Evals Platform
• promptfoo — Open-Source LLM Eval CLI
• Weights & Biases — Experiment Tracking Platform

Slide 14 of 30Open this slide

Model Routing

Send Easy Queries to Small Models, Hard Queries to Large Models

Router Implementation Options

Real-World Impact

A typical support chatbot: 70% simple, 20% medium, 10% complex. With routing: 60–80% cost reduction vs sending everything to a frontier model.

Sources

• RouteLLM: Learning to Route LLMs with Preference Data — arXiv:2406.18665 (Ong et al., 2024)

Slide 15 of 30Open this slide

The Cascade Pattern

Try Small First, Escalate If Quality Is Low

Quality Check Strategies

• Confidence score: model logprob / entropy threshold
• Format validation: does output match expected schema?
• Length heuristic: suspiciously short or long responses
• Self-consistency: generate twice, check agreement
• Lightweight verifier: small classifier trained on good/bad outputs

Economics

If the cascade resolves 80% at tier 1 and 15% at tier 2:

• Only 5% of requests hit the expensive frontier model
• Average cost drops 10–20x compared to frontier-only

Slide 16 of 30Open this slide

Cost Calculator: Monthly Spend Across Model Tiers

Scenario: 10M requests/month, ~500 tokens avg per request (5B tokens/month)

Self-hosted costs assume amortized GPU rental (e.g., A100 at ~$1.50/hr handling ~100 req/s).

The Hybrid Approach

Private-Endpoint Rule of Thumb

At a smaller scale — say 10,000 daily queries — running an SLM on a private endpoint typically costs $500–$2,000/month, versus $5,000–$50,000/month for equivalent frontier API usage. That is roughly a 10–32× cost differential for tasks where the small model holds quality.

Key Insight

A well-designed routing strategy achieves 90%+ of frontier quality at 5–10% of the cost.

Sources

• Small Language Models vs LLMs (Lucas8)

Slide 17 of 30Open this slide

Decision Framework: When to Go Small

The Right-Sizing Ladder

Before picking a model size, decide how much machinery the task actually needs. Climb only as high as you must:

Prompt engineering → RAG → SLM fine-tune → Distill

Start at the bottom rung and stop at the lowest one that meets your quality bar, rather than defaulting to a frontier model. Most tasks are solved well before the top of the ladder.

Step 1: Define Your Quality Bar

• What is the minimum acceptable performance?
• Measure with your own eval set, not public benchmarks
• Get stakeholder agreement on the threshold before testing models

Step 2: Characterize Your Task

• Is it narrow (classification, extraction) or open-ended (creative writing, analysis)?
• Is the output structured (JSON, SQL, labels) or freeform (paragraphs, essays)?
• Do you have labeled training data (1,000+ examples)?
• Narrow + structured + labeled data = strong signal for small models

Step 3: Estimate Your Volume

• Under 10K requests/month? Cost matters less — optimize for quality
• 100K–1M requests/month? Model routing starts to pay off
• Over 1M requests/month? Self-hosting becomes economically viable

Step 4: Test the Smallest Viable Model

• Start with the smallest model in the relevant class
• Run your custom eval suite against it
• If it passes your quality bar, you are done
• If not, move one tier up and repeat

Step 5: Evaluate Deployment Constraints

• Latency requirement under 100ms? Self-hosted small model required
• Data cannot leave your network? On-premise deployment is now a viable production default, not a niche constraint — on-device inference delivers frontier-comparable performance for many tasks, so keeping all inference on-prem rarely costs you quality
• Need offline capability? Edge/on-device model required
• None of the above? API-served models are simplest

Sources

• Best Local LLM Models 2026 (SitePoint)
• The Best Open-Source Small Language Models (BentoML)

Step 6: Implement and Monitor

• Deploy with A/B testing against your current model
• Monitor quality metrics continuously
• Set up alerts for quality degradation
• Keep the larger model as a fallback

Slide 18 of 30Open this slide

Local Inference Tools

Ollama

• One-command setup: ollama run llama3.2:3b
• Manages model downloads, quantization, and serving
• REST API compatible with OpenAI format
• macOS, Linux, Windows support
• Best for: development, prototyping, single-user apps
• Limitation: single-request at a time, no batching

llama.cpp / llama-server

• Pure C/C++ inference — no Python dependency
• GGUF format with extensive quantization options (Q2 through Q8)
• Runs on CPU, Apple Metal, CUDA, Vulkan, SYCL
• Extremely memory efficient — 3B model in ~2GB RAM
• Best for: edge deployment, embedded systems, maximum portability
• Powers many other tools (Ollama uses it internally)

vLLM

• Production-grade serving with PagedAttention
• Continuous batching for high throughput
• OpenAI-compatible API server out of the box
• Tensor parallelism across multiple GPUs
• Best for: production serving at scale, high concurrency
• Throughput: 10–50x over naive HuggingFace inference

Text Generation Inference (TGI)

• HuggingFace's production serving solution
• Flash Attention, quantization (GPTQ, AWQ, EETQ)
• Token streaming and grammar-constrained generation
• Docker-first deployment with GPU support
• Best for: HuggingFace ecosystem integration, Docker workflows
• Powers HuggingFace Inference Endpoints

Sources

• Ollama — Local LLM Runner
• llama.cpp — LLM Inference in C/C++ (Gerganov)
• vLLM — High-Throughput LLM Serving Engine
• vLLM: PagedAttention — arXiv:2309.06180 (Kwon et al., 2023)
• Text Generation Inference — HuggingFace Serving Toolkit

Slide 19 of 30Open this slide

Quantization: Shrinking Models for Deployment

What Is Quantization?

Reducing the precision of model weights from 16-bit floats to 8-bit, 4-bit, or even 2-bit integers. This shrinks memory usage and speeds up inference with minimal quality loss.

Quantization Methods

Memory Impact (Llama 3.1 8B example)

Practical Advice

• Start with Q4 (AWQ or GGUF) — best quality/size trade-off
• Q8 if you have the memory — nearly lossless
• Q2 only for extreme edge — noticeable quality degradation
• Always re-run your eval suite after quantization to measure actual impact

Sources

• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers — arXiv:2210.17323 (Frantar et al., 2022)
• AWQ: Activation-aware Weight Quantization — arXiv:2306.00978 (Lin et al., 2023)
• GGML / GGUF Format (Gerganov)
• bitsandbytes — Accessible LLMs via k-bit Quantization
• llama.cpp — GGUF Quantization Support

Slide 20 of 30Open this slide

Edge Deployment: Mobile, Browser, Embedded

Mobile Deployment

• Apple CoreML / MLX: Optimized for Apple Silicon (iPhone, iPad, Mac)
  • Llama 3.2 3B runs at ~30 tokens/sec on iPhone 15 Pro
  • MLX framework enables rapid prototyping on Mac
• Android / MediaTek / Qualcomm: NPU-accelerated inference
  • Llama 3.2 1B at 50+ tokens/sec on Snapdragon 8 Gen 3
  • ExecuTorch (Meta) for cross-platform mobile deployment
• ONNX Runtime Mobile: Cross-platform, optimized for on-device

Browser-Based Inference

• WebLLM (MLC): Run models directly in the browser via WebGPU
  • Llama 3.2 3B, Phi-3 mini, Gemma 2B all supported
  • No server required — fully client-side
  • ~15 tokens/sec on M1 MacBook in Chrome
• Transformers.js: HuggingFace models in the browser via ONNX/WebGPU
• Use cases: offline apps, privacy-first tools, demos

Embedded and IoT

• Raspberry Pi 5 (8GB): Runs Q4 Llama 3.2 1B at ~5 tokens/sec
• NVIDIA Jetson Orin Nano: Runs 3B–7B models for robotics and vision
• Dedicated inference chips: Hailo, Coral, Groq (for edge scenarios)

Key Constraints

Sources

• MLX — Apple Machine Learning Framework
• ExecuTorch — On-Device AI for PyTorch (Meta)
• ONNX Runtime — Cross-Platform ML Inference (Microsoft)
• WebLLM — In-Browser LLM Inference via WebGPU (MLC AI)
• Transformers.js — ML for the Web (HuggingFace)

Slide 21 of 30Open this slide

Model Serving: Optimizing Throughput and Latency

The Serving Stack

vLLM — The Throughput Champion

• PagedAttention: manages KV cache like virtual memory — near-zero waste
• Continuous batching: new requests join in-flight batches
• Tensor parallelism: split one model across multiple GPUs
• Prefix caching: shared system prompts computed once
• Throughput: 8B model on A100 handles 500–2,000 requests/min

TGI (Text Generation Inference)

• Flash Attention 2 for memory-efficient attention
• Watermark detection for generated text
• Grammar-constrained generation (JSON schema, regex)
• Integrated with HuggingFace Hub — pull any model by name

NVIDIA Triton Inference Server

• Multi-framework: serves PyTorch, TensorRT, ONNX, vLLM backends
• Dynamic batching with configurable strategies
• Model ensembles: chain preprocessing + model + postprocessing
• Best for: organizations already in the NVIDIA ecosystem

Key Serving Metrics

Sources

• vLLM: PagedAttention — arXiv:2309.06180 (Kwon et al., 2023)
• vLLM — High-Throughput LLM Serving Engine
• Text Generation Inference — HuggingFace Serving Toolkit
• NVIDIA Triton Inference Server

Slide 22 of 30Open this slide

Continuous Batching and Speculative Decoding

The Problem with Naive Batching

• Static batching waits for all requests in a batch to finish
• Short requests wait for long ones — wasted GPU cycles
• Utilization drops to 30–50% with variable-length outputs

Continuous Batching (Iteration-Level Scheduling)

• New requests join the batch at every decode step
• Completed requests leave immediately — no waiting
• GPU stays busy with 80–95% utilization
• Implemented in vLLM, TGI, and TensorRT-LLM

Time ──►
GPU:  [A,B,C] [A,B,C,D] [A,C,D] [A,D,E] [D,E,F] ...
       ▲                   ▲        ▲
       B starts            B done   C done, E joins

Speculative Decoding — Use Small Models to Speed Up Big Ones

1. Draft model (small, fast): generates K candidate tokens
2. Target model (large, accurate): verifies all K tokens in one forward pass
3. Accepted tokens are free — rejected tokens cost one extra step

Why It Works

• Large model verification of K tokens costs roughly the same as generating 1
• If draft model matches 70–80% of the time, you get 2–3x speedup
• No quality loss — output is identical to the large model alone

Practical Setup

Combined Impact

Continuous batching + speculative decoding together can yield 3–5x throughput improvement over naive single-request inference.

Sources

• vLLM: PagedAttention — arXiv:2309.06180 (Kwon et al., 2023)
• Fast Inference from Transformers via Speculative Decoding — arXiv:2211.17192 (Leviathan et al., 2023)
• TensorRT-LLM — NVIDIA LLM Inference Library

Slide 23 of 30Open this slide

Hardware Landscape for Small Model Inference

NVIDIA GPUs

• H100/H200: The datacenter workhorse, 80–141 GB HBM3
• A100 (40/80GB): Still widely available, excellent price/perf for serving
• L4 (24GB): Low-power inference GPU, great for 7B–14B models
• RTX 4090 (24GB): Consumer GPU, surprisingly effective for serving
• Cloud: ~$2–4/hr (A100), ~$0.50–1/hr (L4)

Apple Silicon

• M4 Pro/Max/Ultra: Unified memory (36–192 GB) — fits large models
• 7B Q4 at 50–80 tok/s on M4 Pro, 14B Q4 at 30–50 tok/s
• MLX framework optimized for Apple hardware
• Best for: development, prototyping, small-team serving
• Unbeatable power efficiency (watts per token)

Google TPUs

• TPU v5e: Cost-optimized for inference, ~$1.20/hr
• TPU v6e: Next-gen, strong for batch inference workloads
• Tight integration with JAX, Keras, Vertex AI
• Best for: teams already in GCP ecosystem
• Excellent for serving Gemma models natively

Dedicated Inference Accelerators

• Groq LPU: Deterministic latency, 500+ tok/s for small models
• Cerebras WSE: Wafer-scale engine for massive throughput
• AWS Inferentia2: Cost-effective on AWS, ~$0.75/hr
• Intel Gaudi 2/3: Competitive pricing, good for 7B–14B models
• Emerging: Etched Sohu, SambaNova, specialized RISC-V chips

Sources

• MLX — Apple Machine Learning Framework
• TensorRT-LLM — NVIDIA LLM Inference Library

Slide 24 of 30Open this slide

The Hybrid Fleet: Mixing Model Sizes in One Application

Fleet Composition Per Feature

Slide 25 of 30Open this slide

A/B Testing Model Swaps

Why A/B Test Model Changes?

• Offline evals do not capture the full picture of production behavior
• User satisfaction, engagement, and task completion rates matter
• Edge cases in production data may not appear in eval sets
• Stakeholder confidence requires empirical evidence

A/B Test Design for Model Swaps

Traffic Split

• Start with 5–10% of traffic to the new (smaller) model
• Gradually ramp to 25%, 50%, then 100% if metrics hold
• Keep the original model as the control throughout

Metrics to Track

Statistical Rigor

• Minimum sample size: 1,000+ requests per variant
• Run for at least 1–2 weeks to capture weekly patterns
• Use sequential testing to enable early stopping
• Segment by query type — the small model may fail on specific categories

Rollback Plan

• Automated rollback if quality drops below threshold
• Feature flag or traffic percentage toggle — instant switch
• Shadow mode: run both models, compare outputs, serve only one
• Log all inputs/outputs for post-hoc analysis

Slide 26 of 30Open this slide

Migration Strategy: Frontier to Small Models Safely

Step 1: Instrument Your Current System

• Add logging for every LLM call: input, output, model, latency, cost
• Tag requests by feature, user segment, and complexity
• Establish baseline metrics for quality, cost, and latency
• Run for 2+ weeks to capture representative traffic patterns

Step 2: Identify Migration Candidates

• Sort features by cost (highest spend first)
• Flag tasks with structured output (JSON, labels, SQL) — easiest wins
• Flag high-volume, low-complexity tasks — biggest cost savings
• Deprioritize tasks requiring complex reasoning or creativity

Step 3: Build Your Eval Pipeline

• Extract 500+ real production examples per task
• Get ground-truth labels (human annotation or frontier model consensus)
• Automate: run_eval(model, dataset) -> scores
• Include latency and cost in the eval report

Step 4: Test Candidate Models Offline

• Run 3–5 small model candidates through your eval pipeline
• Compare against frontier model baseline
• Identify the smallest model that meets your quality threshold
• Test with quantization to find the optimal size/quality trade-off

Step 5: Shadow Deploy

• Run the small model in parallel with production (shadow mode)
• Compare outputs without serving to users
• Catch regressions before they impact anyone
• Duration: 1–2 weeks of production traffic

Step 6: Gradual Rollout

• 5% traffic to small model with real-time quality monitoring
• Ramp to 25%, 50%, 100% over 2–4 weeks
• Automated rollback triggers on quality drops
• Celebrate the cost savings — then repeat for the next feature

Slide 27 of 30Open this slide

Monitoring Quality Degradation After Model Downsizing

The Silent Failure Problem

Small models fail differently than large models:

• They do not refuse — they confidently produce wrong answers
• Degradation is often gradual, not catastrophic
• Failures cluster on specific input types, not randomly
• Without monitoring, you may not notice for weeks

What to Monitor

Output Quality Signals

• Format compliance: JSON validity rate, schema adherence
• Factual accuracy: spot-check samples against ground truth
• Hallucination rate: claims not grounded in provided context
• Completeness: are outputs truncated or missing fields?
• Consistency: same input should produce similar outputs

Behavioral Signals

• Retry rate: users asking the same question again
• Edit rate: users modifying model outputs before using them
• Abandonment: users leaving the feature without completing their task
• Escalation: users requesting human support after model interaction

Drift Detection

• Monitor input distribution — are users sending different types of queries?
• Track output token length distribution — sudden changes signal problems
• Compare weekly quality scores against baseline
• Use embedding similarity to detect new query clusters the model has not seen

Alerting Strategy

The Feedback Loop

Slide 28 of 30Open this slide

Key Takeaways

The Right Model Is the Smallest One That Works

• Start small, scale up only when quality demands it
• A fine-tuned 7B model often matches frontier models on focused tasks
• Public benchmarks do not predict performance on your specific use case
• Build custom evals before making any model decision

Cost Savings Are Massive and Real

• 100–1,000x cost reduction is achievable for many production tasks
• Model routing and cascade patterns capture 90% of savings with 95% of quality
• Self-hosting becomes economical above 1M requests/month
• The hybrid fleet is the dominant production architecture

The Ecosystem Is Mature

• Llama, Qwen, Phi, Gemma, Mistral — strong options at every size
• Deployment tools (vLLM, llama.cpp, Ollama) are production-ready
• Quantization (Q4) cuts memory in half with minimal quality loss
• Edge deployment on phones and browsers is practical today

Migrate Safely, Monitor Continuously

• Shadow deploy and A/B test every model swap
• Instrument quality metrics from day one
• Small models fail confidently — silent degradation is the real risk
• Keep frontier models as a fallback, not as a default

Slide 29 of 30Open this slide

Key Papers and Benchmarks

Foundational Papers

Inference and Serving

Benchmarks to Know

Recommended Reading Order

1. Llama 3 paper (understand the foundation)
2. Qwen 2.5 / Qwen 3 reports (see what small models can do)
3. vLLM paper (understand production serving)
4. RouteLLM paper (model routing strategies)
5. Run your own benchmarks (the most important step)

Slide 30 of 30Open this slide

The Best Model Is the One You Can Afford to Run

At scale, model selection is an economic decision as much as a technical one.

This Week's Assignment

1. Pick a task from your project (classification, extraction, generation, or Q&A)
2. Build a 200-example eval set from real or realistic data
3. Benchmark at least 3 models: one frontier, one mid-size, one small (<8B)
4. Plot the cost-quality Pareto frontier
5. Write a 1-page recommendation: which model would you deploy and why?

Next week: Fine-Tuning Small Models — LoRA, QLoRA, and making 3B models punch above their weight.