# Week 13: Distillation and fine-tuning

Applied AI Course — Sawan Ruparel

Public self-study edition. See https://applied-ai.sawanruparel.com/updates/ for version notes.

# Knowledge Distillation & Fine-Tuning

## Teaching Small Models Big Tricks

Week 13 | Applied AI Course | Spring 2026

---

*"The goal is not to replace the teacher, but to create a student that excels at the exam."*

---

# Why Distill: Frontier Quality at Small-Model Cost

```mermaid
flowchart LR
    T["Teacher<br/>GPT-4o / Claude Opus<br/>$10-60 / M tokens<br/>500-2000ms"] -->|generates training data| Train[Distillation training]
    Train --> S["Student<br/>Llama 8B / Phi-4<br/>$0.10-0.50 / M tokens<br/>20-100ms"]
    S -->|specialized, 10-100x cheaper| Prod[Production]
```

## Teacher (Frontier Model)

- **GPT-4o, Claude Opus, Gemini Ultra**
- Billions of parameters
- $10-60 per million tokens
- 500-2000ms latency per request
- Broad general knowledge
- Excellent at complex reasoning
- Too expensive for high-volume production

## Student (Distilled Model)

- **Llama 3 8B, Mistral 7B, Phi-3 Mini**
- Millions to low billions of parameters
- $0.10-0.50 per million tokens (self-hosted)
- 20-100ms latency per request
- Specialized for your task
- Matches teacher on narrow domain
- 10-100x cheaper at inference time

---

**The insight:** You don't need a PhD-level generalist to answer your customer support tickets. You need a specialist trained by one.

---

# The Customization Spectrum

```mermaid
flowchart LR
    P[1. Prompting<br/>$0, 0 examples] --> FS[2. Few-shot<br/>5-20 examples]
    FS --> FT[3. Fine-tune LoRA<br/>100-10K examples<br/>$5-50]
    FT --> KD[4. Distillation<br/>teacher + synthetic<br/>$50-500]
    KD --> PT[5. Pre-train<br/>billions of tokens<br/>$10K-10M]
```

## Step 1: Prompting
- Zero engineering effort
- No training data needed
- Limited by context window
- **Cost:** Free (per-token API cost only)
- **When:** Prototyping, simple tasks

## Step 2: Few-Shot / In-Context Learning
- 5-20 examples in the prompt
- No model modification
- Consumes context window tokens
- **Cost:** Slightly higher per-call token cost
- **When:** Consistent formatting, style matching

## Step 3: Fine-Tuning (LoRA/QLoRA)
- 100-10,000 training examples
- Modify model weights with adapters
- Task-specific behavior baked in
- **Cost:** $5-50 in compute, hours of work
- **When:** Domain adaptation, cost reduction

## Step 4: Knowledge Distillation
- Teacher generates synthetic training data
- Student fine-tuned on teacher outputs
- Captures reasoning patterns of larger model
- **Cost:** $50-500 in API + compute
- **When:** Production deployment at scale

## Step 5: Pre-Training from Scratch
- Billions of tokens of raw data
- Train entire model from random weights
- Full control over knowledge and behavior
- **Cost:** $10K-$10M+ in compute
- **When:** Almost never (for most teams)

---

**This week we focus on Steps 3 and 4** -- the sweet spot for applied AI teams.

---

# When Fine-Tuning Beats Prompting

## Signs you need fine-tuning

- **Consistent output format:** JSON schemas, structured extraction, specific templates
- **Domain vocabulary:** Medical, legal, financial terminology the base model fumbles
- **Cost reduction:** Paying $0.10/call when a fine-tuned small model does it for $0.001/call
- **Latency requirements:** Need <100ms responses that large models cannot deliver
- **Privacy constraints:** Data cannot leave your infrastructure

## Signs you should stick with prompting

- Task changes frequently (fine-tuning is a snapshot in time)
- You have fewer than 50 high-quality examples
- The frontier model already does it well and volume is low
- You need broad general knowledge, not narrow specialization

## The decision framework

```mermaid
flowchart TB
    V{Volume > 10K calls/day?}
    V -->|Yes| Q1{Prompting<br/>quality sufficient?}
    Q1 -->|Yes| P1[Stay with prompting<br/>monitor costs]
    Q1 -->|No| FT1[Fine-tune]
    V -->|No| Q2{Latency critical<br/>< 100ms?}
    Q2 -->|Yes| FT2[Fine-tune a small model]
    Q2 -->|No| P2[Prompt engineering + caching]
```

## Rule of thumb

**Fine-tuning ROI = (cost_per_api_call - cost_per_finetuned_call) x daily_volume x 30 - training_cost**

If positive within one month, fine-tuning is worth it.

## Fine-tune vs RAG vs distill (2026 consensus)

These techniques are complementary, not competing -- match the method to the problem:

- **RAG first** for *dynamic knowledge* -- facts that change often or are too large to bake into weights.
- **Fine-tune** for *style, format, and domain vocabulary* that RAG cannot inject through context alone.
- **Distill** when *deploying to constrained edge hardware* where you need the behavior in a small, self-contained model.

## Sources

- [Fine-Tuning LLMs: When RAG Isn't Enough (BigDataBoutique)](https://bigdataboutique.com/blog/fine-tuning-llms-when-rag-isnt-enough)

---

# Synthetic Data: Using Teacher Models to Generate Training Examples

## The core idea

Instead of manually labeling thousands of examples, use a powerful "teacher" model (GPT-4o, Claude Opus) to generate training data for a smaller "student" model.

```mermaid
flowchart LR
    Seed[Seed inputs<br/>topics, schema] --> T[Teacher model<br/>GPT-4o / Claude Opus]
    T --> D[Synthetic examples<br/>10K+ generated overnight]
    D --> Filter[Quality filter]
    Filter --> Train[Train student<br/>Llama 8B / Phi-4]
    Train --> Deploy[Deploy: cheap, fast]
```

## Why synthetic data works

- **Teacher models encode human preferences** -- they were trained on vast human feedback
- **Cheaper than human annotation** -- $0.01/example vs $1-5/example for human labelers
- **Scalable** -- generate 10,000 examples overnight
- **Consistent** -- no inter-annotator disagreement

## Real-world examples

| Project | Teacher | Student | Data Size | Result |
|---------|---------|---------|-----------|--------|
| Alpaca | text-davinci-003 | LLaMA 7B | 52K instructions | Matched teacher on many tasks |
| Orca | GPT-4 | LLaMA 13B | 5M explanations | Beat ChatGPT on benchmarks |
| Phi-1 | GPT-3.5 | 1.3B custom | 1B tokens (textbook) | Matched 10x larger models on code |
| WizardLM | ChatGPT | LLaMA 7B | 250K evolved | Beat Alpaca significantly |

## Key principle

**The student doesn't learn the teacher's knowledge -- it learns the teacher's behavior on your specific task.**

## Sources

- [Alpaca: A Strong, Replicable Instruction-Following Model (Taori et al., 2023)](https://crfm.stanford.edu/2023/03/13/alpaca.html)
- [Orca: Progressive Learning from Complex Explanation Traces of GPT-4 (Mukherjee et al., 2023)](https://arxiv.org/abs/2306.02707)
- [Textbooks Are All You Need / Phi-1 (Gunasekar et al., 2023)](https://arxiv.org/abs/2306.11644)
- [WizardLM: Empowering Large Language Models to Follow Complex Instructions (Xu et al., 2023)](https://arxiv.org/abs/2304.12244)

---

# Synthetic Data Pipeline

```mermaid
flowchart LR
    S1["1. Define task<br/>schema, rubric, failures"] --> S2["2. Seed inputs<br/>real examples, edge cases"]
    S2 --> S3["3. Prompt teacher<br/>system + few-shot, T=0.7-1.0"]
    S3 --> S4["4. Filter & validate<br/>dedupe, quality, length"]
    S4 --> S5["5. Format & split<br/>chat format, 80/10/10"]
    S5 --> S6["6. Train student<br/>LoRA/QLoRA + SFT"]
```

## Practical example: generating customer support data

```python
import openai

SYSTEM_PROMPT = """You are generating training data for a customer support bot.
Given a product and issue category, generate a realistic customer message
and an ideal agent response. Output as JSON:
{"customer": "...", "agent": "...", "category": "...", "sentiment": "..."}"""

SEED_TOPICS = ["billing", "shipping", "returns", "technical", "account"]
PRODUCTS = ["Pro Plan", "Enterprise", "Starter Kit", "API Access"]

for topic in SEED_TOPICS:
    for product in PRODUCTS:
        response = openai.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "system", "content": SYSTEM_PROMPT},
                {"role": "user", "content": f"Product: {product}, Issue: {topic}"}
            ],
            temperature=0.9,  # High diversity
            n=5               # 5 variations per combo
        )
```

---

# Quality Control for Synthetic Data

## The garbage-in-garbage-out problem

Synthetic data amplifies teacher mistakes. A 5% error rate in 10K examples means 500 bad training signals.

## Deduplication

```python
from datasketch import MinHash, MinHashLSH

# Near-duplicate detection with MinHash LSH
lsh = MinHashLSH(threshold=0.8, num_perm=128)

for idx, example in enumerate(dataset):
    mh = MinHash(num_perm=128)
    for word in example["text"].split():
        mh.update(word.encode("utf-8"))
    # Check for near-duplicates before inserting
    if not lsh.query(mh):
        lsh.insert(str(idx), mh)
        clean_dataset.append(example)
```

## Automated filtering

- **Length bounds:** Remove responses shorter than 20 tokens or longer than 2000
- **Format validation:** Parse JSON outputs, reject malformed ones
- **Self-consistency:** Generate 3 responses, keep only when majority agree
- **Perplexity filter:** Remove examples where a reference model is highly uncertain

## LLM-as-judge quality scoring

```python
JUDGE_PROMPT = """Rate this training example 1-5 on:
1. Correctness: Is the response factually accurate?
2. Relevance: Does it address the input?
3. Completeness: Is anything missing?
4. Format: Does it match the required schema?

Only keep examples scoring 4+ on ALL dimensions."""
```

## Human spot-checks

- Review 2-5% of generated data manually
- Focus on edge cases and low-confidence outputs
- Track error categories to fix the generation prompt
- **Budget rule:** 1 hour of human review per 1,000 synthetic examples

---

# Ensuring Diversity: Avoiding Mode Collapse in Synthetic Datasets

## The mode collapse problem

Teacher models have preferred response patterns. Without intervention, your dataset will contain thousands of variations of the same "comfortable" answer.

## Symptoms of low diversity

- Student always starts responses with "Certainly!" or "Of course!"
- Narrow vocabulary compared to desired output
- All responses have similar length and structure
- Poor performance on edge cases and rare categories

## Diversity strategies

### 1. Vary the generation parameters
```python
temperatures = [0.7, 0.8, 0.9, 1.0, 1.1]
top_p_values = [0.9, 0.95, 1.0]
# Generate with different combos for natural variation
```

### 2. Use a topic/scenario taxonomy
- Create an explicit grid: topics x difficulty x format x persona
- Ensure each cell has minimum coverage (e.g., 10 examples per cell)
- Track category distribution and fill gaps

### 3. Prompt perturbation (Evol-Instruct style)
- **Add constraints:** "Respond in under 50 words"
- **Increase complexity:** "Now include error handling"
- **Switch perspective:** "Explain as if to a 5-year-old"
- **Inject edge cases:** "The user is frustrated and swearing"

### 4. Multi-teacher generation
- Use GPT-4o for half, Claude Opus for half
- Each model has different stylistic biases
- The blend produces more robust training data

## Measuring diversity

- **Unique n-gram ratio:** Count distinct trigrams / total trigrams (target >0.7)
- **Topic entropy:** Cluster embeddings, measure uniformity across clusters
- **Response length distribution:** Should match your desired production distribution

## Sources

- [WizardLM: Empowering Large Language Models to Follow Complex Instructions / Evol-Instruct (Xu et al., 2023)](https://arxiv.org/abs/2304.12244)

---

# Generation Techniques

## Instruction Following

Generate input-output pairs where the teacher demonstrates following specific instructions.

```json
{
  "instruction": "Extract all dates mentioned in the text and return them in ISO 8601 format.",
  "input": "The meeting is on March 5th, 2026 and the deadline was January 15.",
  "output": "- 2026-03-05\n- 2026-01-15"
}
```

**Best for:** Classification, extraction, formatting tasks

**Tip:** Vary instruction phrasing -- "Extract dates", "Find all dates", "List the dates mentioned"

## Chain-of-Thought Traces

Capture the teacher's reasoning process, not just the final answer. Teaches the student *how* to think.

```json
{
  "question": "Is 127 prime?",
  "reasoning": "Check divisibility: 127/2=63.5, 127/3=42.3, 127/5=25.4, 127/7=18.1, 127/11=11.5. Since sqrt(127)<12 and no divisor found, 127 is prime.",
  "answer": "Yes, 127 is prime."
}
```

**Best for:** Math, logic, multi-step reasoning

**Tip:** Ask teacher to show work with "Think step by step before answering"

## Tool Use Examples

Teach the student when and how to call external tools (APIs, search, calculators).

```json
{
  "user": "What is the weather in Tokyo right now?",
  "assistant_action": "<tool_call>get_weather(city='Tokyo')</tool_call>",
  "tool_result": "{\"temp\": 18, \"condition\": \"cloudy\"}",
  "assistant_response": "It is currently 18C and cloudy in Tokyo."
}
```

**Best for:** Agentic workflows, RAG systems, function calling

**Tip:** Include examples where the model should NOT use a tool

## Conversation / Multi-Turn

Generate full conversations with realistic back-and-forth, clarification questions, and follow-ups.

```json
{
  "messages": [
    {"role": "user", "content": "Help me write a Python function"},
    {"role": "assistant", "content": "What should the function do?"},
    {"role": "user", "content": "Parse a CSV and return the sum of column 3"},
    {"role": "assistant", "content": "```python\nimport csv\ndef sum_column...```"}
  ]
}
```

**Best for:** Chatbots, customer support, tutoring systems

**Tip:** Include conversations with corrections and misunderstandings

---

# Check Permissions Before Generating Training Data

Permission to call a model API does not establish permission to use every input or output for every downstream purpose. Review the actual service agreement, model license, and dataset terms for the version and use case you choose.

## Record the relevant terms

- The teacher provider, exact model version, agreement, and date checked
- Whether the intended synthetic-data and downstream-training use is permitted
- The base model's license and redistribution requirements
- Permissions for the original examples, documents, and annotations
- Any restrictions on sensitive data, geography, or onward distribution

Do not substitute a general provider name or a claim that a model is open-weight for reading its terms. Seek appropriate review when the intended use is unclear.

## Data quality and ethics

- Preserve provenance: teacher, prompt, sampling settings, filters, and transformations
- Check synthetic data for privacy leakage, bias, and near-duplicates
- Keep development data separate from held-out evaluation data
- Disclose AI assistance and the origin of training examples
- Test whether adding synthetic data helps on independently collected examples

## Reproducibility

Version the dataset and record the model and library versions. Publish only the data you have permission to share. Explain both measured improvements and remaining failures.

Sources

- [Llama model cards and version-specific licenses](https://github.com/meta-llama/llama-models)
- [Primary implementation documentation](/resources/)

---

# Full Fine-Tuning: Update All Parameters

## What it is

Update every weight in the model using your training data. The original pre-trained weights are modified directly.

## The math

For a 7B parameter model:
- **Parameters:** 7,000,000,000 floating point numbers
- **Memory (fp16):** ~14 GB just for weights
- **Memory (training):** ~56 GB (weights + gradients + optimizer states + activations)
- **Hardware:** Typically requires 2-4x A100 80GB GPUs

## Advantages

- Maximum expressiveness -- can change any behavior
- No architectural constraints on what can be learned
- Historically the gold standard for task adaptation

## Disadvantages

- **Catastrophic forgetting:** Model loses general capabilities while learning your task
- **Storage:** Need to save a full copy of the model per task (14 GB per fine-tune)
- **Cost:** GPU hours add up fast -- $2-10/hour for A100s
- **Overfitting risk:** With small datasets (<1K examples), full fine-tuning easily overfits

## When to use full fine-tuning

- You have a large dataset (>50K examples)
- You need to fundamentally change model behavior
- You have significant compute budget
- You are training a foundation model for further specialization

## In practice: most teams skip this and go straight to LoRA

Full fine-tuning is the "from scratch" renovation. LoRA is the "targeted remodel."

---

# LoRA: Low-Rank Adaptation

## The key insight

Most of the "knowledge" learned during fine-tuning lives in a low-dimensional subspace. You don't need to update all 7 billion parameters -- a small set of adapter matrices can capture task-specific behavior.

## How it works (intuition)

1. **Freeze** the entire pre-trained model (no gradients flow to original weights)
2. **Add** small trainable matrices alongside specific layers (usually attention layers)
3. **Train** only these adapter matrices (~0.1-1% of total parameters)
4. **Merge** adapters back into base model for deployment (zero inference overhead)

## By the numbers

| Model Size | Full Fine-Tune Params | LoRA Params (rank 16) | Reduction |
|------------|----------------------|----------------------|-----------|
| 7B | 7,000M | 4-20M | 350-1750x |
| 13B | 13,000M | 8-40M | 325-1625x |
| 70B | 70,000M | 40-200M | 350-1750x |

## Practical benefits

- **Train on a single GPU:** A 7B model with LoRA fits in ~10 GB VRAM
- **Fast training:** Minutes to hours, not days
- **No catastrophic forgetting:** Base model knowledge is preserved
- **Stackable:** Swap different LoRA adapters for different tasks
- **Tiny storage:** Each adapter is 10-100 MB, not 14 GB

## LoRA targets

Common layers to apply LoRA to:
- `q_proj`, `v_proj` -- minimum effective set (attention queries and values)
- `q_proj`, `k_proj`, `v_proj`, `o_proj` -- full attention (recommended)
- All linear layers -- maximum expressiveness, higher cost

## KD-LoRA: a hybrid with distillation

**KD-LoRA** combines LoRA adapters with a knowledge-distillation signal from a teacher model -- the adapter is trained against the teacher's outputs rather than hard labels alone. The result is performance comparable to full fine-tuning at much lower resource cost, bridging the LoRA and distillation tracks of this course.

## Sources

- [LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)](https://arxiv.org/abs/2106.09685)
- [KD-LoRA: A Hybrid Approach Combining LoRA with Knowledge Distillation (arXiv:2410.20777)](https://arxiv.org/abs/2410.20777)

---

# How LoRA Works: Low-Rank Weight Decomposition

```mermaid
flowchart LR
    X[Input x] --> W["W (d x d)<br/>FROZEN<br/>~16M params"]
    X --> A["A (d x r)<br/>trainable<br/>~256K"]
    A --> B["B (r x d)<br/>trainable<br/>~256K"]
    W --> Sum((+))
    B --> Scale["× alpha/rank"]
    Scale --> Sum
    Sum --> Out[Output]
```

```
  Forward pass:   output = W @ x  +  (B @ A) @ x * (alpha / rank)

  Where:
    d = hidden dimension (e.g., 4096)
    r = rank (e.g., 16, 32, 64)  << much smaller than d
    alpha = scaling factor (typically 2 * rank)
```

## Why "low-rank"?

```
  Full weight update:     delta_W  is  4096 x 4096  = 16,777,216 parameters
  LoRA approximation:     B @ A    is  4096 x 16 x 16 x 4096 = 131,072 + 131,072
                                                                = 262,144 parameters

  That is a 64x reduction for rank=16!
```

## Initialization matters

```python
# A is initialized with random Gaussian values
nn.init.kaiming_uniform_(self.lora_A)

# B is initialized to ZERO
nn.init.zeros_(self.lora_B)

# This means: at the start of training, the adapter has NO effect
# output = W @ x + (0 @ A) @ x = W @ x
# Training gradually learns the delta from the pre-trained behavior
```

## The scaling factor: alpha / rank

- `alpha` controls how much the adapter influences the output
- Common setting: `alpha = 2 * rank` (so scaling = 2.0)
- Higher alpha = adapter has more influence = faster learning but less stable
- Lower alpha = more conservative updates = more stable but slower

## Sources

- [LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)](https://arxiv.org/abs/2106.09685)

---

# QLoRA: Fine-Tune on a Single GPU

## The breakthrough

QLoRA combines two techniques to make fine-tuning dramatically more memory efficient:

1. **Quantize** the base model to 4-bit (NF4 format) -- reduces memory 4x
2. **Add LoRA adapters** in full precision (bf16/fp16) -- maintains training quality

## Memory comparison for Llama 3 8B

| Method | VRAM Required | Hardware |
|--------|--------------|----------|
| Full fine-tune (fp16) | ~60 GB | 1x A100 80GB |
| LoRA (fp16 base) | ~18 GB | 1x A100 40GB |
| QLoRA (4-bit base) | ~6 GB | 1x RTX 4090 or even RTX 3090 |
| QLoRA + gradient checkpointing | ~4.5 GB | 1x RTX 3080 |

## The NF4 data type

- "Normal Float 4-bit" -- quantization levels are optimized for normally-distributed weights
- Double quantization: quantize the quantization constants too
- Paged optimizers: offload optimizer states to CPU when GPU memory is full

## Code example

```python
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model

# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

# Load quantized base model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)

# Add LoRA adapters
lora_config = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 13,631,488 || all params: 8,043,847,680 || trainable%: 0.1695
```

## Quality vs full fine-tuning

QLoRA matches full fine-tuning quality on most tasks, with <1% degradation on benchmarks. The original paper showed this across 1,000+ experiments.

## What this costs in practice

Using **Unsloth + QLoRA**, a single **A100 80GB** fine-tunes **Llama 3 8B in ~6 hours on 50k examples for ~$12**. Unsloth's extended-context GRPO pushes this further: up to **110K context on an 80GB H100** -- about **7x longer** than vanilla training at the same memory budget.

## Sources

- [QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., 2023)](https://arxiv.org/abs/2305.14314)
- [LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)](https://arxiv.org/abs/2106.09685)
- [Unsloth vs Traditional Fine-Tuning: Faster GRPO Training Explained (YottaLabs)](https://www.yottalabs.ai/post/unsloth-vs-traditional-fine-tuning-faster-grpo-training-explained)
- [Hugging Face PEFT](https://github.com/huggingface/peft)

---

# Training Setup: Dataset Format, Hyperparameters, Hardware

## Dataset format: ChatML / conversation format

```json
{
  "messages": [
    {"role": "system", "content": "You are a helpful medical coding assistant."},
    {"role": "user", "content": "What ICD-10 code is used for type 2 diabetes?"},
    {"role": "assistant", "content": "The ICD-10 code for type 2 diabetes mellitus is E11. More specific codes include:\n- E11.0: With hyperosmolarity\n- E11.65: With hyperglycemia\n- E11.9: Without complications (most common)"}
  ]
}
```

Save as JSONL -- one JSON object per line, one conversation per line.

## Dataset size guidelines

| Task Complexity | Minimum Examples | Recommended | Typical Quality |
|----------------|-----------------|-------------|-----------------|
| Simple classification | 50 | 500-1,000 | Good |
| Structured extraction | 200 | 1,000-5,000 | Good |
| Conversational style | 500 | 2,000-10,000 | Good |
| Complex reasoning | 1,000 | 5,000-50,000 | Varies |

## Hardware options and cost

| GPU | VRAM | Method | 8B Model | Approx. Cost |
|-----|------|--------|----------|-------------|
| RTX 3090/4090 | 24 GB | QLoRA | Yes | $0.50/hr (cloud) |
| A100 40GB | 40 GB | LoRA | Yes | $1.50/hr (cloud) |
| A100 80GB | 80 GB | Full FT | Yes | $2.50/hr (cloud) |
| H100 80GB | 80 GB | Full FT | Yes (faster) | $3.50/hr (cloud) |

## Cloud options

- **Lambda Labs:** Cheapest on-demand A100s (~$1.10/hr)
- **RunPod:** Flexible spot pricing, good for experiments
- **Google Colab Pro+:** A100 for $50/month (limited hours)
- **AWS/GCP/Azure:** Most expensive but most reliable

## Typical training time

- **1K examples, QLoRA, 8B model:** 15-30 minutes on A100
- **10K examples, QLoRA, 8B model:** 2-4 hours on A100
- **50K examples, LoRA, 8B model:** 8-16 hours on A100

---

# Supervised Fine-Tuning (SFT)

## What is SFT?

Train the model to produce specific outputs given specific inputs. The simplest and most common form of fine-tuning.

## The training objective

```
Loss = -sum( log P(output_token_i | input, output_tokens_<i) )
```

The model learns to maximize the probability of the desired output tokens, given the input context. This is the same causal language modeling loss used in pre-training, but applied only to the assistant's response tokens.

## Masking: only train on the response

```python
# Input:  <system>You are helpful.</system><user>What is 2+2?</user><assistant>4</assistant>
# Labels: [-100, -100, -100, ..., -100, -100, ..., 4, </assistant>]
#          ^^^^ system tokens masked  ^^^^ user tokens masked  ^^^ only train here

# -100 is PyTorch's ignore_index -- these tokens don't contribute to the loss
```

This is critical: you want the model to learn the *response*, not to memorize the *prompt*.

## SFT with TRL (Transformers Reinforcement Learning)

```python
from trl import SFTTrainer, SFTConfig

training_config = SFTConfig(
    output_dir="./output",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,   # Effective batch size = 16
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    bf16=True,
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=100,
    save_strategy="steps",
    save_steps=100,
    max_seq_length=2048,
)

trainer = SFTTrainer(
    model=model,
    args=training_config,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    processing_class=tokenizer,
)
trainer.train()
```

## When SFT is enough

- Consistent output formatting (JSON, markdown, specific schemas)
- Domain-specific language and terminology
- Instruction following for a narrow task
- Style transfer (formal, casual, brand voice)

## Sources

- [Hugging Face TRL](https://github.com/huggingface/trl)

---

# DPO, GRPO and RLHF: Preference Alignment

## Beyond SFT: teaching what humans prefer

SFT teaches the model *what* to say. Preference alignment teaches it *which response is better* when multiple answers are possible.

## RLHF (Reinforcement Learning from Human Feedback)

1. Train a **reward model** on human preference data (A is better than B)
2. Use PPO (Proximal Policy Optimization) to optimize the LLM against the reward model
3. Complex, unstable, requires significant engineering

By 2026, **DPO has largely displaced RLHF in production** -- it is cheaper, more stable, and comparable in quality. Hugging Face **TRL v1.0** (April 2026) reflects this shift, treating DPO, GRPO, ORPO, and reward modeling as first-class, unified post-training pipelines rather than bolt-on scripts.

## DPO (Direct Preference Optimization) -- the simpler alternative

Skip the reward model entirely. Train directly on preference pairs.

```json
{
  "prompt": "Explain quantum computing simply.",
  "chosen": "Think of regular computers as coins that are heads or tails. Quantum computers are like spinning coins -- they can be both at once until you look.",
  "rejected": "Quantum computing utilizes quantum mechanical phenomena such as superposition and entanglement to perform computational operations on data, leveraging qubits instead of classical bits."
}
```

## DPO training with TRL

```python
from trl import DPOTrainer, DPOConfig

dpo_config = DPOConfig(
    output_dir="./dpo-output",
    num_train_epochs=1,           # DPO typically needs fewer epochs
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=5e-7,           # Much lower LR than SFT
    beta=0.1,                     # KL penalty -- controls how far from base model
    bf16=True,
    max_length=2048,
    max_prompt_length=1024,
)

trainer = DPOTrainer(
    model=model,
    ref_model=None,  # Uses implicit reference (PEFT adapter as ref)
    args=dpo_config,
    train_dataset=preference_dataset,
    processing_class=tokenizer,
)
trainer.train()
```

## GRPO (Group Relative Policy Optimization)

Introduced with **DeepSeek R1**, GRPO removes both the reward model *and* the value model entirely. Instead of scoring against a learned reward model, it ranks a *group* of sampled responses against each other and optimizes relative to the group mean. The payoff: lower memory footprint and faster training than PPO-style RLHF. GRPO is supported natively in **Unsloth** and **Hugging Face TRL v1.0** (April 2026).

## Typical pipeline

```mermaid
flowchart LR
    Base[Base model] --> SFT[SFT<br/>learn the task]
    SFT --> DPO[DPO / GRPO<br/>refine preferences]
    DPO --> Deploy[Deploy]
```

## When to add DPO

- Model produces correct answers but with poor style or tone
- You need to reduce harmful or undesirable outputs
- Multiple valid responses exist and you want to prefer one style
- You have (or can generate) preference pairs from your teacher model

## Sources

- [DPO: Direct Preference Optimization (Rafailov et al., 2023)](https://arxiv.org/abs/2305.18290)
- [Training Language Models to Follow Instructions with Human Feedback / InstructGPT (Ouyang et al., 2022)](https://arxiv.org/abs/2203.02155)
- [Reinforcement Learning (RL) Guide — GRPO (Unsloth Docs)](https://unsloth.ai/docs/get-started/reinforcement-learning-rl-guide)
- [Hugging Face Releases TRL v1.0: Unified Post-Training Stack for SFT, Reward Modeling, DPO, GRPO (MarkTechPost)](https://www.marktechpost.com/2026/04/01/hugging-face-releases-trl-v1-0-a-unified-post-training-stack-for-sft-reward-modeling-dpo-and-grpo-workflows/)
- [LLM Fine-Tuning Best Practices (HJ Labs)](https://hjlabs.in/AIML/blog/post/llm-fine-tuning-best-practices.html)
- [Hugging Face TRL](https://github.com/huggingface/trl)

---

# Fine-Tuning Tools

## Hugging Face TRL

The standard library for SFT, DPO, and RLHF. Built on top of Transformers.

```bash
pip install trl peft transformers datasets
```

- **Pros:** Well-documented, active community, integrates with HF Hub
- **Cons:** Requires more boilerplate, manual config
- **Best for:** Custom pipelines, research, full control

```python
from trl import SFTTrainer, SFTConfig
# Full control over every parameter
```

## Axolotl

YAML-driven fine-tuning framework. Configure everything in a config file, run one command.

```bash
accelerate launch -m axolotl.cli.train config.yml
```

- **Pros:** Minimal code, many recipes, multi-GPU support
- **Cons:** Less flexible for custom training loops
- **Best for:** Practitioners who want fast iteration

```yaml
base_model: meta-llama/Llama-3.1-8B-Instruct
adapter: qlora
lora_r: 16
```

## Unsloth

Optimized for speed. Claims 2-5x faster training with 60% less memory through custom CUDA kernels.

```bash
pip install unsloth
```

- **Pros:** Fastest training, free Colab tier support
- **Cons:** Supports fewer model architectures
- **Best for:** Quick experiments, limited hardware

```python
from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
    "unsloth/Llama-3.1-8B-Instruct", load_in_4bit=True
)
```

## OpenAI Fine-Tuning API

Managed fine-tuning service. Upload data, click train, get an endpoint.

```bash
openai api fine_tuning.jobs.create \
  -t "training_data.jsonl" -m "gpt-4o-mini-2024-07-18"
```

- **Pros:** Zero infrastructure, production-ready endpoint
- **Cons:** Limited model choices, no weight access, ongoing API costs
- **Best for:** Teams without ML infrastructure, production deployments

## Sources

- [Hugging Face TRL](https://github.com/huggingface/trl)
- [Hugging Face PEFT](https://github.com/huggingface/peft)
- [Axolotl](https://github.com/axolotl-ai-cloud/axolotl)
- [Unsloth](https://github.com/unslothai/unsloth)
- [OpenAI Fine-Tuning Docs](https://platform.openai.com/docs/guides/fine-tuning)

---

# Dataset Preparation: Conversation Format

## The standard format: messages array

Every major framework expects data in the "messages" conversation format:

```json
{"messages": [
  {"role": "system", "content": "You are a sentiment analysis API. Respond with exactly one word: positive, negative, or neutral."},
  {"role": "user", "content": "I absolutely love this product! Best purchase ever."},
  {"role": "assistant", "content": "positive"}
]}
```

## Converting raw data to conversation format

```python
import json

def convert_to_chat(row):
    """Convert a classification dataset row to chat format."""
    return {
        "messages": [
            {
                "role": "system",
                "content": "Classify the customer review sentiment as positive, negative, or neutral. Respond with only the label."
            },
            {"role": "user", "content": row["review_text"]},
            {"role": "assistant", "content": row["label"]}
        ]
    }

# Write as JSONL
with open("train.jsonl", "w") as f:
    for row in dataset:
        f.write(json.dumps(convert_to_chat(row)) + "\n")
```

## Multi-turn conversations

```json
{"messages": [
  {"role": "system", "content": "You are a Python tutor."},
  {"role": "user", "content": "How do I read a CSV file?"},
  {"role": "assistant", "content": "Use pandas: `import pandas as pd; df = pd.read_csv('file.csv')`"},
  {"role": "user", "content": "How do I filter rows where age > 30?"},
  {"role": "assistant", "content": "`filtered = df[df['age'] > 30]`"}
]}
```

## Data splits

```python
from sklearn.model_selection import train_test_split

# 80% train, 10% validation, 10% test
train, temp = train_test_split(data, test_size=0.2, random_state=42)
val, test = train_test_split(temp, test_size=0.5, random_state=42)
```

## Pre-flight checklist

- [ ] All examples parse as valid JSON
- [ ] No empty assistant responses
- [ ] System prompt is consistent across examples (or intentionally varied)
- [ ] Max sequence length covers 95%+ of your examples
- [ ] No PII or sensitive data leaked from seed inputs
- [ ] Train/val/test sets have no overlap

---

# The Training Loop: Setup, Train, Evaluate, Iterate

```mermaid
flowchart LR
    S1[1. Setup<br/>quantize + LoRA] --> S2[2. Train<br/>SFTTrainer]
    S2 --> S3[3. Evaluate<br/>vs baseline]
    S3 --> D{Pass?}
    D -->|No| S4[4. Iterate<br/>more data, tune HP]
    S4 --> S2
    D -->|Yes| Deploy[Deploy]
```

## Step 1: Setup

Load base model with quantization. Add LoRA adapters. Prepare tokenizer.

```python
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    max_seq_length=2048,
    load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model(
    model, r=16, lora_alpha=32,
    target_modules=["q_proj","k_proj","v_proj","o_proj",
                     "gate_proj","up_proj","down_proj"],
    lora_dropout=0.05,
)
```

## Step 2: Train

Launch training with early stopping on validation loss.

```python
from trl import SFTTrainer, SFTConfig

trainer = SFTTrainer(
    model=model,
    train_dataset=train_data,
    eval_dataset=val_data,
    args=SFTConfig(
        output_dir="./checkpoints",
        num_train_epochs=3,
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        learning_rate=2e-4,
        warmup_ratio=0.05,
        eval_strategy="steps",
        eval_steps=50,
        save_strategy="steps",
        save_steps=50,
        load_best_model_at_end=True,
        bf16=True,
    ),
    processing_class=tokenizer,
)
trainer.train()
```

## Step 3: Evaluate

Run inference on held-out test set. Compare against baseline (base model + prompting).

```python
# Generate predictions on test set
results = []
for example in test_data:
    prompt = tokenizer.apply_chat_template(
        example["messages"][:-1], tokenize=False, add_generation_prompt=True
    )
    output = model.generate(tokenizer(prompt, return_tensors="pt").input_ids.cuda(),
                            max_new_tokens=256, temperature=0.1)
    prediction = tokenizer.decode(output[0], skip_special_tokens=True)
    results.append({"expected": example["messages"][-1]["content"],
                     "predicted": prediction})
```

## Step 4: Iterate

Analyze failures. Add more training data for weak categories. Adjust hyperparameters. Repeat.

- If overfitting: reduce epochs, increase dropout, add more data
- If underfitting: increase rank, increase learning rate, train longer
- If format errors: add more format-specific examples
- If hallucinating: add negative examples, reduce temperature at inference

## Sources

- [Unsloth](https://github.com/unslothai/unsloth)
- [Hugging Face TRL](https://github.com/huggingface/trl)

---

# Key Hyperparameters

## The ones that matter most

| Parameter | Typical Range | Recommended Start | What it controls |
|-----------|--------------|-------------------|------------------|
| **Learning rate** | 1e-5 to 5e-4 | 2e-4 | Speed of weight updates |
| **LoRA rank (r)** | 4 to 128 | 16 | Capacity of adapter (expressiveness) |
| **LoRA alpha** | 8 to 256 | 32 (2x rank) | Scaling of adapter contribution |
| **Epochs** | 1 to 5 | 3 | Passes through the data |
| **Batch size** | 1 to 32 | 4 (x4 grad accum = 16 effective) |Stability of gradient estimates |
| **LoRA dropout** | 0.0 to 0.1 | 0.05 | Regularization |
| **Max seq length** | 512 to 8192 | 2048 | Longest example to process |
| **Warmup ratio** | 0.0 to 0.1 | 0.05 | Gradual LR increase at start |

## Learning rate: the most important hyperparameter

```
Too high (>5e-4):  Training loss oscillates, model produces garbage
Too low  (<1e-5):  Model barely changes from base, wasted compute
Sweet spot (2e-4): Standard for LoRA with AdamW optimizer
```

- Use **cosine** learning rate schedule (gradual decay)
- Always include a **warmup** phase (5% of total steps)

## LoRA rank: capacity vs efficiency tradeoff

```
Rank 4:    Very lightweight, good for simple tasks (classification)
Rank 16:   Good default for most tasks
Rank 32:   More capacity, needed for complex generation
Rank 64:   Near full fine-tune expressiveness
Rank 128:  Rarely needed, diminishing returns
```

## Practical tips

- **Start small, scale up:** Begin with rank=16, lr=2e-4, 3 epochs
- **Watch validation loss:** If it starts increasing, you are overfitting
- **Effective batch size matters:** `batch_size x gradient_accumulation_steps x num_gpus`
- **Gradient checkpointing:** Trade compute for memory -- enable it if running out of VRAM
- **bf16 > fp16:** bfloat16 is more numerically stable, use it when hardware supports it

---

# Evaluating Fine-Tuned Models

## Two types of evaluation

### 1. Benchmark evals (general capability)

Check that your fine-tuned model has not lost general abilities:

- **MMLU:** General knowledge across 57 subjects
- **HumanEval:** Code generation
- **GSM8K:** Grade-school math reasoning
- **HellaSwag:** Commonsense reasoning

Run these with `lm-evaluation-harness`:

```bash
lm_eval --model hf \
    --model_args pretrained=./my-finetuned-model \
    --tasks mmlu,hellaswag,gsm8k \
    --batch_size 8
```

### 2. Task-specific evals (your actual use case)

These matter far more than benchmarks. Design evaluations that mirror production usage.

```python
# Example: evaluating a JSON extraction model
def evaluate_extraction(model, test_set):
    metrics = {"exact_match": 0, "valid_json": 0, "field_accuracy": {}}

    for example in test_set:
        prediction = generate(model, example["input"])

        # Can it produce valid JSON?
        try:
            parsed = json.loads(prediction)
            metrics["valid_json"] += 1
        except json.JSONDecodeError:
            continue

        # Does it match expected output?
        if parsed == example["expected"]:
            metrics["exact_match"] += 1

        # Field-level accuracy
        for field in example["expected"]:
            if field not in metrics["field_accuracy"]:
                metrics["field_accuracy"][field] = {"correct": 0, "total": 0}
            metrics["field_accuracy"][field]["total"] += 1
            if parsed.get(field) == example["expected"][field]:
                metrics["field_accuracy"][field]["correct"] += 1

    return metrics
```

## The evaluation matrix

| What to Measure | How | Target |
|----------------|-----|--------|
| Format compliance | Parse output, check schema | >98% |
| Task accuracy | Compare to gold labels | >90% (task dependent) |
| Latency | Time per inference | <100ms for prod |
| General capability | Benchmark suite | <5% drop from base |
| Robustness | Adversarial / edge case inputs | No catastrophic failures |

## Always compare against baselines

1. Base model + zero-shot prompt
2. Base model + few-shot prompt
3. Teacher model (upper bound)
4. Your fine-tuned model

If your fine-tuned model does not beat baseline #2, something is wrong.

## Sources

- [lm-evaluation-harness (EleutherAI)](https://github.com/EleutherAI/lm-evaluation-harness)

---

# Common Failures in Fine-Tuning

## 1. Catastrophic forgetting

**Symptom:** Model excels at your task but cannot do anything else. Ask it a general question and it responds with task-specific output.

**Cause:** Too many epochs, learning rate too high, or dataset too narrow.

**Fix:**
- Reduce to 1-3 epochs
- Lower learning rate (try 1e-4 instead of 2e-4)
- Mix in 10-20% general instruction data (e.g., Alpaca or OpenAssistant)
- Use LoRA instead of full fine-tuning

## 2. Overfitting

**Symptom:** Training loss drops to near zero but validation loss increases. Model memorizes training examples verbatim.

**Cause:** Too few unique examples, too many epochs, rank too high.

**Fix:**
- Add more diverse training data
- Reduce epochs (watch validation loss curve)
- Increase LoRA dropout to 0.1
- Reduce LoRA rank
- Use early stopping: `load_best_model_at_end=True`

## 3. Format collapse

**Symptom:** Model ignores output format requirements. Asked for JSON, responds in prose. Asked for bullets, gives paragraphs.

**Cause:** Insufficient format-specific examples, or format examples not diverse enough.

**Fix:**
- Ensure every training example demonstrates the correct format
- Add explicit format instructions in the system prompt
- Include "negative" examples: malformed format -> correct format
- Increase the proportion of format-heavy examples

## 4. Reward hacking (DPO/RLHF)

**Symptom:** Model produces verbose, sycophantic responses that score well on preference metrics but are not actually helpful.

**Cause:** Preference data is biased toward longer or more agreeable responses.

**Fix:**
- Include preference pairs where concise answers are preferred
- Add length-controlled pairs (short preferred over long for simple questions)
- Lower the DPO beta to constrain divergence from the base model

## 5. Repetition loops

**Symptom:** Model gets stuck repeating phrases or sentences.

**Cause:** Repetitive training data, or training on outputs with repetition.

**Fix:**
- Deduplicate training data aggressively
- Use repetition penalty at inference time: `repetition_penalty=1.1`
- Filter training data for examples containing repetition

## Debugging checklist

```
[ ] Validation loss is decreasing (not just training loss)
[ ] Model can still answer general questions
[ ] Output format matches expectations on >95% of test cases
[ ] No verbatim training examples in outputs (memorization check)
[ ] Performance beats base model + prompting baseline
```

## Sources

- [Alpaca: A Strong, Replicable Instruction-Following Model (Taori et al., 2023)](https://crfm.stanford.edu/2023/03/13/alpaca.html)
- [OpenAssistant (LAION-AI)](https://github.com/LAION-AI/Open-Assistant)
- [LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)](https://arxiv.org/abs/2106.09685)
- [DPO: Direct Preference Optimization (Rafailov et al., 2023)](https://arxiv.org/abs/2305.18290)

---

# End-to-End Distillation Pipeline

```mermaid
flowchart LR
    T["Teacher model<br/>GPT-4o / Claude Opus"] --> G["Synthetic data<br/>5K-50K filtered examples"]
    G --> ST["Student training<br/>Llama 8B + QLoRA"]
    ST --> Merge["Merge adapter<br/>Base + LoRA"]
    Merge --> Eval["Evaluate<br/>task evals + benchmarks"]
    Eval --> Deploy["Deploy<br/>vLLM / TGI / Ollama"]
    Eval -->|Fails bar| G
```

## Phase details with cost estimates

| Phase | Time | Cost | Key Decision |
|-------|------|------|-------------|
| Define task & seed data | 1-2 days | $0 | What exact behavior do you need? |
| Teacher generation | 4-12 hours | $50-500 (API) | How many examples? Quality vs quantity |
| Filtering & validation | 2-4 hours | $20-50 (judge LLM) | What quality threshold? |
| Student training | 1-4 hours | $5-20 (GPU) | What rank/epochs? |
| Evaluation | 2-4 hours | $0 (local compute) | Does it beat the baseline? |
| Merge & deploy | 30 min | $0 | Merge adapter or serve separately? |

## Total cost for a typical project

```
Teacher API calls (10K examples):     $100-300
Quality filtering (LLM judge):         $20-50
GPU compute (QLoRA training):           $5-20
Evaluation compute:                     $5-10
                                      ----------
Total:                                 $130-380

vs. running the teacher model in production:
10K calls/day x $0.01/call x 30 days = $3,000/month

ROI: Pays for itself in ~4 days
```

---

# Iterative Distillation: Refine Based on Student Failures

## The key insight

Your first distillation pass will not be perfect. The student will fail on edge cases, rare categories, and inputs that differ from the training distribution. Use these failures to improve.

## The iterative loop

```mermaid
flowchart LR
    R1[Round 1<br/>5K examples] --> Train1[Train student]
    Train1 --> Eval1[Evaluate]
    Eval1 --> Fail[Mine failure cases]
    Fail --> R2[Round 2+<br/>targeted examples<br/>for failure modes]
    R2 --> Train1
    Eval1 --> Done{Quality bar met?}
    Done -->|Yes| Ship[Ship]
```

## Mining failures for targeted data generation

```python
# After evaluation, collect failure cases
failures = [ex for ex in test_results if ex["correct"] == False]

# Categorize failures
failure_categories = {}
for f in failures:
    category = classify_failure(f)  # e.g., "long_input", "ambiguous", "multi_label"
    failure_categories.setdefault(category, []).append(f)

# Generate targeted synthetic data for each failure category
for category, examples in failure_categories.items():
    prompt = f"""Generate 50 training examples similar to these failure cases.
    The model struggled with: {category}
    Example failures:
    {json.dumps(examples[:5], indent=2)}

    Generate diverse examples that cover this failure mode."""

    new_data = teacher.generate(prompt)
```

## Curriculum learning approach

- **Round 1:** Easy, clear-cut examples (build basic competency)
- **Round 2:** Medium difficulty, add ambiguity and edge cases
- **Round 3:** Hard examples, adversarial inputs, rare categories
- **Round 4:** Mix of all difficulties (reinforce and stabilize)

## When to stop iterating

- Task-specific eval accuracy plateaus (less than 1% improvement per round)
- You have hit the student model's capacity ceiling
- Diminishing returns on teacher API spending
- The student matches or exceeds your quality threshold

## Practical tip: the "hard example" multiplier

If the student fails on 5% of cases, generate 10x more training examples for those cases relative to easy cases. This re-balances the training distribution toward the model's weak spots.

---

# Model Merging: Combining Multiple Fine-Tunes

## The idea

You have multiple LoRA adapters (or full fine-tunes) specialized for different tasks. Merge them into a single model that can do everything.

## Merging methods

### TIES (Trim, Elect Sign, Merge)
- Trims small weight changes (noise)
- Resolves sign conflicts between models
- Merges remaining weights by averaging
- **Best for:** Merging 3+ models with overlapping capabilities

### DARE (Drop And REscale)
- Randomly drops a fraction of weight changes
- Rescales remaining weights to compensate
- Reduces interference between models
- **Best for:** Merging models with conflicting specializations

### SLERP (Spherical Linear Interpolation)
- Interpolates between two models along a spherical path
- Preserves the magnitude of weight vectors
- Single interpolation parameter `t` controls the blend (0.0 to 1.0)
- **Best for:** Smooth blending of two models

## Using mergekit

```bash
pip install mergekit

# merge_config.yml
models:
  - model: ./base-model
    parameters:
      weight: 1.0
  - model: ./lora-customer-support
    parameters:
      weight: 0.5
  - model: ./lora-code-generation
    parameters:
      weight: 0.3

merge_method: ties
parameters:
  density: 0.5      # TIES: fraction of weights to keep
  normalize: true
dtype: bfloat16
```

```bash
mergekit-yaml merge_config.yml ./merged-output --cuda
```

## When to merge vs. serve separately

| Scenario | Approach |
|----------|----------|
| Tasks are related (support + sales) | Merge |
| Tasks are unrelated (code + medical) | Serve separately |
| Need to update one task frequently | Serve separately |
| Want single deployment simplicity | Merge |
| Memory constrained | Merge |

## Warning: merging is not guaranteed to work

- Always evaluate the merged model on all tasks
- Quality often drops 5-15% compared to individual adapters
- Iterative merging with evaluation is essential

## Sources

- [TIES-Merging: Resolving Interference When Merging Models (Yadav et al., 2023)](https://arxiv.org/abs/2306.01708)
- [Language Models are Super Mario / DARE (Yu et al., 2023)](https://arxiv.org/abs/2311.03099)
- [mergekit (Arcee AI)](https://github.com/arcee-ai/mergekit)

---

# Serving LoRA Adapters at Runtime

## The multi-adapter pattern

Keep one base model in memory. Swap LoRA adapters per request for different tasks.

```mermaid
flowchart LR
    Req[Request] --> R{Router by task}
    R --> L1[LoRA: Customer Support]
    R --> L2[LoRA: Code Review]
    R --> L3[LoRA: Medical Triage]
    R --> L4[LoRA: Legal Summary]
    L1 --> Base[(Base model 8B<br/>shared in memory)]
    L2 --> Base
    L3 --> Base
    L4 --> Base
    Base --> Out[Response]
```

**Memory:** Base model (4.5 GB quantized) + each adapter (~20 MB) = serve 50 tasks in 5.5 GB

## vLLM with LoRA adapters

```python
from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest

# Load base model once
llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    enable_lora=True,
    max_lora_rank=64,
    max_loras=4,  # Max concurrent adapters in memory
)

# Serve different tasks by swapping adapters
support_request = LoRARequest("support", 1, "./lora-support")
code_request = LoRARequest("code", 2, "./lora-code")

# Route based on task
output = llm.generate(
    "Help me with my billing issue",
    SamplingParams(temperature=0.3, max_tokens=512),
    lora_request=support_request,
)
```

## Ollama with custom models

```bash
# Create a Modelfile
cat > Modelfile <<EOF
FROM llama3.1:8b
ADAPTER ./my-lora-adapter.gguf
SYSTEM "You are a customer support agent for Acme Corp."
PARAMETER temperature 0.3
EOF

ollama create acme-support -f Modelfile
ollama run acme-support "I need to cancel my subscription"
```

## Production serving checklist

- [ ] Adapter hot-loading (no server restart to add new adapters)
- [ ] Request routing (which adapter for which request)
- [ ] Adapter version management (A/B test new adapters)
- [ ] Fallback to base model if adapter fails
- [ ] Latency monitoring per adapter
- [ ] Adapter caching strategy (LRU eviction for memory management)

## Adapter versioning strategy

```
/adapters/
  /customer-support/
    v1.0.0/  (initial training)
    v1.1.0/  (added edge cases)
    v2.0.0/  (retrained on new product line)
  /code-review/
    v1.0.0/
```

Tag adapters like software releases. Roll back if quality regresses.

## Sources

- [vLLM: High-Throughput LLM Serving](https://github.com/vllm-project/vllm)
- [Ollama](https://github.com/ollama/ollama)
- [LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)](https://arxiv.org/abs/2106.09685)

---

# Key Takeaways

## Distillation is a Superpower

Use frontier models as teachers to generate high-quality synthetic training data. The student does not need to match the teacher everywhere -- just on your specific task. This is the most cost-effective way to deploy AI at scale.

## LoRA Makes Fine-Tuning Accessible

You do not need a cluster of A100s. QLoRA lets you fine-tune an 8B model on a single consumer GPU in under an hour. The barrier to fine-tuning is now data quality, not compute.

## Data Quality > Data Quantity

1,000 excellent examples beat 50,000 mediocre ones. Invest in filtering, deduplication, diversity, and human spot-checks. Your model is only as good as its training data.

## Evaluate Like You Deploy

Benchmarks tell you if the model is broken. Task-specific evals tell you if it is useful. Always compare against the baselines: base model + prompting, and the teacher model itself.

## Iterate Relentlessly

First-pass distillation rarely achieves target quality. Mine failures, generate targeted examples, retrain, and repeat. Budget for 2-4 rounds of refinement.

## Ship Adapters, Not Models

Serve a single base model with swappable LoRA adapters. Each adapter is 20 MB, version-controlled, and independently updatable. This is the deployment pattern for multi-task production systems.

---

# Key Papers and Resources

## Essential reading

| Paper | Year | Key Contribution |
|-------|------|-----------------|
| **[LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685)** (Hu et al.) | 2021 | Introduced low-rank adapters for parameter-efficient fine-tuning |
| **[QLoRA: Efficient Finetuning of Quantized LLMs](https://arxiv.org/abs/2305.14314)** (Dettmers et al.) | 2023 | 4-bit quantization + LoRA, enabling single-GPU fine-tuning |
| **[Self-Instruct](https://arxiv.org/abs/2212.10560)** (Wang et al.) | 2022 | Framework for generating instruction data using a language model |
| **[Alpaca](https://crfm.stanford.edu/2023/03/13/alpaca.html)** (Taori et al., Stanford) | 2023 | 52K instruction-following examples from text-davinci-003 to fine-tune LLaMA |
| **[Orca: Progressive Learning from Complex Explanation Traces](https://arxiv.org/abs/2306.02707)** (Mukherjee et al.) | 2023 | Fine-tuning on GPT-4 explanation traces, showing reasoning transfer |
| **[Phi-1: Textbooks Are All You Need](https://arxiv.org/abs/2306.11644)** (Gunasekar et al.) | 2023 | Synthetic "textbook-quality" data to train small but capable models |
| **[WizardLM: Empowering Large Language Models to Follow Complex Instructions](https://arxiv.org/abs/2304.12244)** | 2023 | Evol-Instruct for generating progressively complex training data |
| **[DPO: Direct Preference Optimization](https://arxiv.org/abs/2305.18290)** (Rafailov et al.) | 2023 | Simplified preference alignment without a reward model |

## Practical resources

- **[Hugging Face PEFT](https://github.com/huggingface/peft):** Comprehensive guide to LoRA, QLoRA, and other adapters
- **[TRL (Transformer Reinforcement Learning)](https://github.com/huggingface/trl):** SFT, DPO, PPO trainers
- **[Axolotl](https://github.com/axolotl-ai-cloud/axolotl):** YAML-driven fine-tuning with many model support
- **[Unsloth](https://github.com/unslothai/unsloth):** Optimized training kernels for fast LoRA fine-tuning
- **[mergekit](https://github.com/arcee-ai/mergekit):** Model merging toolkit (TIES, DARE, SLERP)
- **[lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness):** Standardized benchmark evaluation
- **[vLLM](https://github.com/vllm-project/vllm):** High-performance inference with LoRA adapter support

## Community hubs

- **Hugging Face Hub:** Model hosting, datasets, and spaces for demos
- **Open LLM Leaderboard:** Compare model performance across benchmarks
- **r/LocalLLaMA:** Community discussion on fine-tuning and deployment
- **Alignment Forum:** Research discussion on preference learning and safety

---

# Week 13 Wrap-Up

## You now know how to:

1. Generate synthetic training data using teacher models
2. Fine-tune with LoRA and QLoRA on a single GPU
3. Evaluate and iterate on distilled models
4. Deploy adapters in production

---

## Next week: Safety & Red-Teaming

Test prompt injection and agent hijacking, apply layered controls, and measure remaining failures.

---

*"The best model is not the biggest one -- it is the one that ships."*