Distillation and fine-tuning

Slide 1 of 30Open this slide

Knowledge Distillation & Fine-Tuning

Teaching Small Models Big Tricks

Week 13 | Applied AI Course | Spring 2026

"The goal is not to replace the teacher, but to create a student that excels at the exam."

Slide 2 of 30Open this slide

Why Distill: Frontier Quality at Small-Model Cost

Teacher (Frontier Model)

• GPT-4o, Claude Opus, Gemini Ultra
• Billions of parameters
• $10-60 per million tokens
• 500-2000ms latency per request
• Broad general knowledge
• Excellent at complex reasoning
• Too expensive for high-volume production

Student (Distilled Model)

• Llama 3 8B, Mistral 7B, Phi-3 Mini
• Millions to low billions of parameters
• $0.10-0.50 per million tokens (self-hosted)
• 20-100ms latency per request
• Specialized for your task
• Matches teacher on narrow domain
• 10-100x cheaper at inference time

The insight: You don't need a PhD-level generalist to answer your customer support tickets. You need a specialist trained by one.

Slide 3 of 30Open this slide

The Customization Spectrum

Step 1: Prompting

• Zero engineering effort
• No training data needed
• Limited by context window
• Cost: Free (per-token API cost only)
• When: Prototyping, simple tasks

Step 2: Few-Shot / In-Context Learning

• 5-20 examples in the prompt
• No model modification
• Consumes context window tokens
• Cost: Slightly higher per-call token cost
• When: Consistent formatting, style matching

Step 3: Fine-Tuning (LoRA/QLoRA)

• 100-10,000 training examples
• Modify model weights with adapters
• Task-specific behavior baked in
• Cost: $5-50 in compute, hours of work
• When: Domain adaptation, cost reduction

Step 4: Knowledge Distillation

• Teacher generates synthetic training data
• Student fine-tuned on teacher outputs
• Captures reasoning patterns of larger model
• Cost: $50-500 in API + compute
• When: Production deployment at scale

Step 5: Pre-Training from Scratch

• Billions of tokens of raw data
• Train entire model from random weights
• Full control over knowledge and behavior
• Cost: $10K-$10M+ in compute
• When: Almost never (for most teams)

This week we focus on Steps 3 and 4 -- the sweet spot for applied AI teams.

Slide 4 of 30Open this slide

When Fine-Tuning Beats Prompting

Signs you need fine-tuning

• Consistent output format: JSON schemas, structured extraction, specific templates
• Domain vocabulary: Medical, legal, financial terminology the base model fumbles
• Cost reduction: Paying $0.10/call when a fine-tuned small model does it for $0.001/call
• Latency requirements: Need <100ms responses that large models cannot deliver
• Privacy constraints: Data cannot leave your infrastructure

Signs you should stick with prompting

• Task changes frequently (fine-tuning is a snapshot in time)
• You have fewer than 50 high-quality examples
• The frontier model already does it well and volume is low
• You need broad general knowledge, not narrow specialization

The decision framework

Rule of thumb

Fine-tuning ROI = (cost_per_api_call - cost_per_finetuned_call) x daily_volume x 30 - training_cost

If positive within one month, fine-tuning is worth it.

Fine-tune vs RAG vs distill (2026 consensus)

These techniques are complementary, not competing -- match the method to the problem:

• RAG first for dynamic knowledge -- facts that change often or are too large to bake into weights.
• Fine-tune for style, format, and domain vocabulary that RAG cannot inject through context alone.
• Distill when deploying to constrained edge hardware where you need the behavior in a small, self-contained model.

Sources

• Fine-Tuning LLMs: When RAG Isn't Enough (BigDataBoutique)

Slide 5 of 30Open this slide

Synthetic Data: Using Teacher Models to Generate Training Examples

The core idea

Instead of manually labeling thousands of examples, use a powerful "teacher" model (GPT-4o, Claude Opus) to generate training data for a smaller "student" model.

Why synthetic data works

• Teacher models encode human preferences -- they were trained on vast human feedback
• Cheaper than human annotation -- $0.01/example vs $1-5/example for human labelers
• Scalable -- generate 10,000 examples overnight
• Consistent -- no inter-annotator disagreement

Real-world examples

Key principle

The student doesn't learn the teacher's knowledge -- it learns the teacher's behavior on your specific task.

Sources

• Alpaca: A Strong, Replicable Instruction-Following Model (Taori et al., 2023)
• Orca: Progressive Learning from Complex Explanation Traces of GPT-4 (Mukherjee et al., 2023)
• Textbooks Are All You Need / Phi-1 (Gunasekar et al., 2023)
• WizardLM: Empowering Large Language Models to Follow Complex Instructions (Xu et al., 2023)

Slide 6 of 30Open this slide

Synthetic Data Pipeline

Practical example: generating customer support data

import openai

SYSTEM_PROMPT = """You are generating training data for a customer support bot.
Given a product and issue category, generate a realistic customer message
and an ideal agent response. Output as JSON:
{"customer": "...", "agent": "...", "category": "...", "sentiment": "..."}"""

SEED_TOPICS = ["billing", "shipping", "returns", "technical", "account"]
PRODUCTS = ["Pro Plan", "Enterprise", "Starter Kit", "API Access"]

for topic in SEED_TOPICS:
    for product in PRODUCTS:
        response = openai.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "system", "content": SYSTEM_PROMPT},
                {"role": "user", "content": f"Product: {product}, Issue: {topic}"}
            ],
            temperature=0.9,  # High diversity
            n=5               # 5 variations per combo
        )

Slide 7 of 30Open this slide

Quality Control for Synthetic Data

The garbage-in-garbage-out problem

Synthetic data amplifies teacher mistakes. A 5% error rate in 10K examples means 500 bad training signals.

Deduplication

from datasketch import MinHash, MinHashLSH

# Near-duplicate detection with MinHash LSH
lsh = MinHashLSH(threshold=0.8, num_perm=128)

for idx, example in enumerate(dataset):
    mh = MinHash(num_perm=128)
    for word in example["text"].split():
        mh.update(word.encode("utf-8"))
    # Check for near-duplicates before inserting
    if not lsh.query(mh):
        lsh.insert(str(idx), mh)
        clean_dataset.append(example)

Automated filtering

• Length bounds: Remove responses shorter than 20 tokens or longer than 2000
• Format validation: Parse JSON outputs, reject malformed ones
• Self-consistency: Generate 3 responses, keep only when majority agree
• Perplexity filter: Remove examples where a reference model is highly uncertain

LLM-as-judge quality scoring

JUDGE_PROMPT = """Rate this training example 1-5 on:
1. Correctness: Is the response factually accurate?
2. Relevance: Does it address the input?
3. Completeness: Is anything missing?
4. Format: Does it match the required schema?

Only keep examples scoring 4+ on ALL dimensions."""

Human spot-checks

• Review 2-5% of generated data manually
• Focus on edge cases and low-confidence outputs
• Track error categories to fix the generation prompt
• Budget rule: 1 hour of human review per 1,000 synthetic examples

Slide 8 of 30Open this slide

Ensuring Diversity: Avoiding Mode Collapse in Synthetic Datasets

The mode collapse problem

Teacher models have preferred response patterns. Without intervention, your dataset will contain thousands of variations of the same "comfortable" answer.

Symptoms of low diversity

• Student always starts responses with "Certainly!" or "Of course!"
• Narrow vocabulary compared to desired output
• All responses have similar length and structure
• Poor performance on edge cases and rare categories

Diversity strategies

1. Vary the generation parameters

temperatures = [0.7, 0.8, 0.9, 1.0, 1.1]
top_p_values = [0.9, 0.95, 1.0]
# Generate with different combos for natural variation

2. Use a topic/scenario taxonomy

• Create an explicit grid: topics x difficulty x format x persona
• Ensure each cell has minimum coverage (e.g., 10 examples per cell)
• Track category distribution and fill gaps

3. Prompt perturbation (Evol-Instruct style)

• Add constraints: "Respond in under 50 words"
• Increase complexity: "Now include error handling"
• Switch perspective: "Explain as if to a 5-year-old"
• Inject edge cases: "The user is frustrated and swearing"

4. Multi-teacher generation

• Use GPT-4o for half, Claude Opus for half
• Each model has different stylistic biases
• The blend produces more robust training data

Measuring diversity

• Unique n-gram ratio: Count distinct trigrams / total trigrams (target >0.7)
• Topic entropy: Cluster embeddings, measure uniformity across clusters
• Response length distribution: Should match your desired production distribution

Sources

• WizardLM: Empowering Large Language Models to Follow Complex Instructions / Evol-Instruct (Xu et al., 2023)

Slide 9 of 30Open this slide

Generation Techniques

Instruction Following

Generate input-output pairs where the teacher demonstrates following specific instructions.

{
  "instruction": "Extract all dates mentioned in the text and return them in ISO 8601 format.",
  "input": "The meeting is on March 5th, 2026 and the deadline was January 15.",
  "output": "- 2026-03-05\n- 2026-01-15"
}

Best for: Classification, extraction, formatting tasks

Tip: Vary instruction phrasing -- "Extract dates", "Find all dates", "List the dates mentioned"

Chain-of-Thought Traces

Capture the teacher's reasoning process, not just the final answer. Teaches the student how to think.

{
  "question": "Is 127 prime?",
  "reasoning": "Check divisibility: 127/2=63.5, 127/3=42.3, 127/5=25.4, 127/7=18.1, 127/11=11.5. Since sqrt(127)<12 and no divisor found, 127 is prime.",
  "answer": "Yes, 127 is prime."
}

Best for: Math, logic, multi-step reasoning

Tip: Ask teacher to show work with "Think step by step before answering"

Tool Use Examples

Teach the student when and how to call external tools (APIs, search, calculators).

{
  "user": "What is the weather in Tokyo right now?",
  "assistant_action": "<tool_call>get_weather(city='Tokyo')</tool_call>",
  "tool_result": "{\"temp\": 18, \"condition\": \"cloudy\"}",
  "assistant_response": "It is currently 18C and cloudy in Tokyo."
}

Best for: Agentic workflows, RAG systems, function calling

Tip: Include examples where the model should NOT use a tool

Conversation / Multi-Turn

Generate full conversations with realistic back-and-forth, clarification questions, and follow-ups.

{
  "messages": [
    {"role": "user", "content": "Help me write a Python function"},
    {"role": "assistant", "content": "What should the function do?"},
    {"role": "user", "content": "Parse a CSV and return the sum of column 3"},
    {"role": "assistant", "content": "```python\nimport csv\ndef sum_column...```"}
  ]
}

Best for: Chatbots, customer support, tutoring systems

Tip: Include conversations with corrections and misunderstandings

Slide 10 of 30Open this slide

Check Permissions Before Generating Training Data

Permission to call a model API does not establish permission to use every input or output for every downstream purpose. Review the actual service agreement, model license, and dataset terms for the version and use case you choose.

Record the relevant terms

• The teacher provider, exact model version, agreement, and date checked
• Whether the intended synthetic-data and downstream-training use is permitted
• The base model's license and redistribution requirements
• Permissions for the original examples, documents, and annotations
• Any restrictions on sensitive data, geography, or onward distribution

Do not substitute a general provider name or a claim that a model is open-weight for reading its terms. Seek appropriate review when the intended use is unclear.

Data quality and ethics

• Preserve provenance: teacher, prompt, sampling settings, filters, and transformations
• Check synthetic data for privacy leakage, bias, and near-duplicates
• Keep development data separate from held-out evaluation data
• Disclose AI assistance and the origin of training examples
• Test whether adding synthetic data helps on independently collected examples

Reproducibility

Version the dataset and record the model and library versions. Publish only the data you have permission to share. Explain both measured improvements and remaining failures.

Sources

• Llama model cards and version-specific licenses
• Primary implementation documentation

Slide 11 of 30Open this slide

Full Fine-Tuning: Update All Parameters

What it is

Update every weight in the model using your training data. The original pre-trained weights are modified directly.

The math

For a 7B parameter model:

• Parameters: 7,000,000,000 floating point numbers
• Memory (fp16): ~14 GB just for weights
• Memory (training): ~56 GB (weights + gradients + optimizer states + activations)
• Hardware: Typically requires 2-4x A100 80GB GPUs

Advantages

• Maximum expressiveness -- can change any behavior
• No architectural constraints on what can be learned
• Historically the gold standard for task adaptation

Disadvantages

• Catastrophic forgetting: Model loses general capabilities while learning your task
• Storage: Need to save a full copy of the model per task (14 GB per fine-tune)
• Cost: GPU hours add up fast -- $2-10/hour for A100s
• Overfitting risk: With small datasets (<1K examples), full fine-tuning easily overfits

When to use full fine-tuning

• You have a large dataset (>50K examples)
• You need to fundamentally change model behavior
• You have significant compute budget
• You are training a foundation model for further specialization

In practice: most teams skip this and go straight to LoRA

Full fine-tuning is the "from scratch" renovation. LoRA is the "targeted remodel."

Slide 12 of 30Open this slide

LoRA: Low-Rank Adaptation

The key insight

Most of the "knowledge" learned during fine-tuning lives in a low-dimensional subspace. You don't need to update all 7 billion parameters -- a small set of adapter matrices can capture task-specific behavior.

How it works (intuition)

1. Freeze the entire pre-trained model (no gradients flow to original weights)
2. Add small trainable matrices alongside specific layers (usually attention layers)
3. Train only these adapter matrices (~0.1-1% of total parameters)
4. Merge adapters back into base model for deployment (zero inference overhead)

By the numbers

Practical benefits

• Train on a single GPU: A 7B model with LoRA fits in ~10 GB VRAM
• Fast training: Minutes to hours, not days
• No catastrophic forgetting: Base model knowledge is preserved
• Stackable: Swap different LoRA adapters for different tasks
• Tiny storage: Each adapter is 10-100 MB, not 14 GB

LoRA targets

Common layers to apply LoRA to:

• q_proj, v_proj -- minimum effective set (attention queries and values)
• q_proj, k_proj, v_proj, o_proj -- full attention (recommended)
• All linear layers -- maximum expressiveness, higher cost

KD-LoRA: a hybrid with distillation

KD-LoRA combines LoRA adapters with a knowledge-distillation signal from a teacher model -- the adapter is trained against the teacher's outputs rather than hard labels alone. The result is performance comparable to full fine-tuning at much lower resource cost, bridging the LoRA and distillation tracks of this course.

Sources

• LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)
• KD-LoRA: A Hybrid Approach Combining LoRA with Knowledge Distillation (arXiv:2410.20777)

Slide 13 of 30Open this slide

How LoRA Works: Low-Rank Weight Decomposition

  Forward pass:   output = W @ x  +  (B @ A) @ x * (alpha / rank)

  Where:
    d = hidden dimension (e.g., 4096)
    r = rank (e.g., 16, 32, 64)  << much smaller than d
    alpha = scaling factor (typically 2 * rank)

Why "low-rank"?

  Full weight update:     delta_W  is  4096 x 4096  = 16,777,216 parameters
  LoRA approximation:     B @ A    is  4096 x 16 x 16 x 4096 = 131,072 + 131,072
                                                                = 262,144 parameters

  That is a 64x reduction for rank=16!

Initialization matters

# A is initialized with random Gaussian values
nn.init.kaiming_uniform_(self.lora_A)

# B is initialized to ZERO
nn.init.zeros_(self.lora_B)

# This means: at the start of training, the adapter has NO effect
# output = W @ x + (0 @ A) @ x = W @ x
# Training gradually learns the delta from the pre-trained behavior

The scaling factor: alpha / rank

• alpha controls how much the adapter influences the output
• Common setting: alpha = 2 * rank (so scaling = 2.0)
• Higher alpha = adapter has more influence = faster learning but less stable
• Lower alpha = more conservative updates = more stable but slower

Sources

• LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)

Slide 14 of 30Open this slide

QLoRA: Fine-Tune on a Single GPU

The breakthrough

QLoRA combines two techniques to make fine-tuning dramatically more memory efficient:

1. Quantize the base model to 4-bit (NF4 format) -- reduces memory 4x
2. Add LoRA adapters in full precision (bf16/fp16) -- maintains training quality

Memory comparison for Llama 3 8B

The NF4 data type

• "Normal Float 4-bit" -- quantization levels are optimized for normally-distributed weights
• Double quantization: quantize the quantization constants too
• Paged optimizers: offload optimizer states to CPU when GPU memory is full

Code example

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model

# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

# Load quantized base model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)

# Add LoRA adapters
lora_config = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 13,631,488 || all params: 8,043,847,680 || trainable%: 0.1695

Quality vs full fine-tuning

QLoRA matches full fine-tuning quality on most tasks, with <1% degradation on benchmarks. The original paper showed this across 1,000+ experiments.

What this costs in practice

Using Unsloth + QLoRA, a single A100 80GB fine-tunes Llama 3 8B in ~6 hours on 50k examples for ~$12. Unsloth's extended-context GRPO pushes this further: up to 110K context on an 80GB H100 -- about 7x longer than vanilla training at the same memory budget.

Sources

• QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., 2023)
• LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)
• Unsloth vs Traditional Fine-Tuning: Faster GRPO Training Explained (YottaLabs)
• Hugging Face PEFT

Slide 15 of 30Open this slide

Training Setup: Dataset Format, Hyperparameters, Hardware

Dataset format: ChatML / conversation format

{
  "messages": [
    {"role": "system", "content": "You are a helpful medical coding assistant."},
    {"role": "user", "content": "What ICD-10 code is used for type 2 diabetes?"},
    {"role": "assistant", "content": "The ICD-10 code for type 2 diabetes mellitus is E11. More specific codes include:\n- E11.0: With hyperosmolarity\n- E11.65: With hyperglycemia\n- E11.9: Without complications (most common)"}
  ]
}

Save as JSONL -- one JSON object per line, one conversation per line.

Dataset size guidelines

Hardware options and cost

Cloud options

• Lambda Labs: Cheapest on-demand A100s (~$1.10/hr)
• RunPod: Flexible spot pricing, good for experiments
• Google Colab Pro+: A100 for $50/month (limited hours)
• AWS/GCP/Azure: Most expensive but most reliable

Typical training time

• 1K examples, QLoRA, 8B model: 15-30 minutes on A100
• 10K examples, QLoRA, 8B model: 2-4 hours on A100
• 50K examples, LoRA, 8B model: 8-16 hours on A100

Slide 16 of 30Open this slide

Supervised Fine-Tuning (SFT)

What is SFT?

Train the model to produce specific outputs given specific inputs. The simplest and most common form of fine-tuning.

The training objective

Loss = -sum( log P(output_token_i | input, output_tokens_<i) )

The model learns to maximize the probability of the desired output tokens, given the input context. This is the same causal language modeling loss used in pre-training, but applied only to the assistant's response tokens.

Masking: only train on the response

# Input:  <system>You are helpful.</system><user>What is 2+2?</user><assistant>4</assistant>
# Labels: [-100, -100, -100, ..., -100, -100, ..., 4, </assistant>]
#          ^^^^ system tokens masked  ^^^^ user tokens masked  ^^^ only train here

# -100 is PyTorch's ignore_index -- these tokens don't contribute to the loss

This is critical: you want the model to learn the response, not to memorize the prompt.

SFT with TRL (Transformers Reinforcement Learning)

from trl import SFTTrainer, SFTConfig

training_config = SFTConfig(
    output_dir="./output",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,   # Effective batch size = 16
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    bf16=True,
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=100,
    save_strategy="steps",
    save_steps=100,
    max_seq_length=2048,
)

trainer = SFTTrainer(
    model=model,
    args=training_config,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    processing_class=tokenizer,
)
trainer.train()

When SFT is enough

• Consistent output formatting (JSON, markdown, specific schemas)
• Domain-specific language and terminology
• Instruction following for a narrow task
• Style transfer (formal, casual, brand voice)

Sources

• Hugging Face TRL

Slide 17 of 30Open this slide

DPO, GRPO and RLHF: Preference Alignment

Beyond SFT: teaching what humans prefer

SFT teaches the model what to say. Preference alignment teaches it which response is better when multiple answers are possible.

RLHF (Reinforcement Learning from Human Feedback)

1. Train a reward model on human preference data (A is better than B)
2. Use PPO (Proximal Policy Optimization) to optimize the LLM against the reward model
3. Complex, unstable, requires significant engineering

By 2026, DPO has largely displaced RLHF in production -- it is cheaper, more stable, and comparable in quality. Hugging Face TRL v1.0 (April 2026) reflects this shift, treating DPO, GRPO, ORPO, and reward modeling as first-class, unified post-training pipelines rather than bolt-on scripts.

DPO (Direct Preference Optimization) -- the simpler alternative

Skip the reward model entirely. Train directly on preference pairs.

{
  "prompt": "Explain quantum computing simply.",
  "chosen": "Think of regular computers as coins that are heads or tails. Quantum computers are like spinning coins -- they can be both at once until you look.",
  "rejected": "Quantum computing utilizes quantum mechanical phenomena such as superposition and entanglement to perform computational operations on data, leveraging qubits instead of classical bits."
}

DPO training with TRL

from trl import DPOTrainer, DPOConfig

dpo_config = DPOConfig(
    output_dir="./dpo-output",
    num_train_epochs=1,           # DPO typically needs fewer epochs
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=5e-7,           # Much lower LR than SFT
    beta=0.1,                     # KL penalty -- controls how far from base model
    bf16=True,
    max_length=2048,
    max_prompt_length=1024,
)

trainer = DPOTrainer(
    model=model,
    ref_model=None,  # Uses implicit reference (PEFT adapter as ref)
    args=dpo_config,
    train_dataset=preference_dataset,
    processing_class=tokenizer,
)
trainer.train()

GRPO (Group Relative Policy Optimization)

Introduced with DeepSeek R1, GRPO removes both the reward model and the value model entirely. Instead of scoring against a learned reward model, it ranks a group of sampled responses against each other and optimizes relative to the group mean. The payoff: lower memory footprint and faster training than PPO-style RLHF. GRPO is supported natively in Unsloth and Hugging Face TRL v1.0 (April 2026).

Typical pipeline

When to add DPO

• Model produces correct answers but with poor style or tone
• You need to reduce harmful or undesirable outputs
• Multiple valid responses exist and you want to prefer one style
• You have (or can generate) preference pairs from your teacher model

Sources

• DPO: Direct Preference Optimization (Rafailov et al., 2023)
• Training Language Models to Follow Instructions with Human Feedback / InstructGPT (Ouyang et al., 2022)
• Reinforcement Learning (RL) Guide — GRPO (Unsloth Docs)
• Hugging Face Releases TRL v1.0: Unified Post-Training Stack for SFT, Reward Modeling, DPO, GRPO (MarkTechPost)
• LLM Fine-Tuning Best Practices (HJ Labs)
• Hugging Face TRL

Slide 18 of 30Open this slide

Fine-Tuning Tools

Hugging Face TRL

The standard library for SFT, DPO, and RLHF. Built on top of Transformers.

pip install trl peft transformers datasets

• Pros: Well-documented, active community, integrates with HF Hub
• Cons: Requires more boilerplate, manual config
• Best for: Custom pipelines, research, full control

from trl import SFTTrainer, SFTConfig
# Full control over every parameter

Axolotl

YAML-driven fine-tuning framework. Configure everything in a config file, run one command.

accelerate launch -m axolotl.cli.train config.yml

• Pros: Minimal code, many recipes, multi-GPU support
• Cons: Less flexible for custom training loops
• Best for: Practitioners who want fast iteration

base_model: meta-llama/Llama-3.1-8B-Instruct
adapter: qlora
lora_r: 16

Unsloth

Optimized for speed. Claims 2-5x faster training with 60% less memory through custom CUDA kernels.

pip install unsloth

• Pros: Fastest training, free Colab tier support
• Cons: Supports fewer model architectures
• Best for: Quick experiments, limited hardware

from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
    "unsloth/Llama-3.1-8B-Instruct", load_in_4bit=True
)

OpenAI Fine-Tuning API

Managed fine-tuning service. Upload data, click train, get an endpoint.

openai api fine_tuning.jobs.create \
  -t "training_data.jsonl" -m "gpt-4o-mini-2024-07-18"

• Pros: Zero infrastructure, production-ready endpoint
• Cons: Limited model choices, no weight access, ongoing API costs
• Best for: Teams without ML infrastructure, production deployments

Sources

• Hugging Face TRL
• Hugging Face PEFT
• Axolotl
• Unsloth
• OpenAI Fine-Tuning Docs

Slide 19 of 30Open this slide

Dataset Preparation: Conversation Format

The standard format: messages array

Every major framework expects data in the "messages" conversation format:

{"messages": [
  {"role": "system", "content": "You are a sentiment analysis API. Respond with exactly one word: positive, negative, or neutral."},
  {"role": "user", "content": "I absolutely love this product! Best purchase ever."},
  {"role": "assistant", "content": "positive"}
]}

Converting raw data to conversation format

import json

def convert_to_chat(row):
    """Convert a classification dataset row to chat format."""
    return {
        "messages": [
            {
                "role": "system",
                "content": "Classify the customer review sentiment as positive, negative, or neutral. Respond with only the label."
            },
            {"role": "user", "content": row["review_text"]},
            {"role": "assistant", "content": row["label"]}
        ]
    }

# Write as JSONL
with open("train.jsonl", "w") as f:
    for row in dataset:
        f.write(json.dumps(convert_to_chat(row)) + "\n")

Multi-turn conversations

{"messages": [
  {"role": "system", "content": "You are a Python tutor."},
  {"role": "user", "content": "How do I read a CSV file?"},
  {"role": "assistant", "content": "Use pandas: `import pandas as pd; df = pd.read_csv('file.csv')`"},
  {"role": "user", "content": "How do I filter rows where age > 30?"},
  {"role": "assistant", "content": "`filtered = df[df['age'] > 30]`"}
]}

Data splits

from sklearn.model_selection import train_test_split

# 80% train, 10% validation, 10% test
train, temp = train_test_split(data, test_size=0.2, random_state=42)
val, test = train_test_split(temp, test_size=0.5, random_state=42)

Pre-flight checklist

• All examples parse as valid JSON
• No empty assistant responses
• System prompt is consistent across examples (or intentionally varied)
• Max sequence length covers 95%+ of your examples
• No PII or sensitive data leaked from seed inputs
• Train/val/test sets have no overlap

Slide 20 of 30Open this slide

The Training Loop: Setup, Train, Evaluate, Iterate

Step 1: Setup

Load base model with quantization. Add LoRA adapters. Prepare tokenizer.

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    max_seq_length=2048,
    load_in_4bit=True,
)
model = FastLanguageModel.get_peft_model(
    model, r=16, lora_alpha=32,
    target_modules=["q_proj","k_proj","v_proj","o_proj",
                     "gate_proj","up_proj","down_proj"],
    lora_dropout=0.05,
)

Step 2: Train

Launch training with early stopping on validation loss.

from trl import SFTTrainer, SFTConfig

trainer = SFTTrainer(
    model=model,
    train_dataset=train_data,
    eval_dataset=val_data,
    args=SFTConfig(
        output_dir="./checkpoints",
        num_train_epochs=3,
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        learning_rate=2e-4,
        warmup_ratio=0.05,
        eval_strategy="steps",
        eval_steps=50,
        save_strategy="steps",
        save_steps=50,
        load_best_model_at_end=True,
        bf16=True,
    ),
    processing_class=tokenizer,
)
trainer.train()

Step 3: Evaluate

Run inference on held-out test set. Compare against baseline (base model + prompting).

# Generate predictions on test set
results = []
for example in test_data:
    prompt = tokenizer.apply_chat_template(
        example["messages"][:-1], tokenize=False, add_generation_prompt=True
    )
    output = model.generate(tokenizer(prompt, return_tensors="pt").input_ids.cuda(),
                            max_new_tokens=256, temperature=0.1)
    prediction = tokenizer.decode(output[0], skip_special_tokens=True)
    results.append({"expected": example["messages"][-1]["content"],
                     "predicted": prediction})

Step 4: Iterate

Analyze failures. Add more training data for weak categories. Adjust hyperparameters. Repeat.

• If overfitting: reduce epochs, increase dropout, add more data
• If underfitting: increase rank, increase learning rate, train longer
• If format errors: add more format-specific examples
• If hallucinating: add negative examples, reduce temperature at inference

Sources

• Unsloth
• Hugging Face TRL

Slide 21 of 30Open this slide

Key Hyperparameters

The ones that matter most

Learning rate: the most important hyperparameter

Too high (>5e-4):  Training loss oscillates, model produces garbage
Too low  (<1e-5):  Model barely changes from base, wasted compute
Sweet spot (2e-4): Standard for LoRA with AdamW optimizer

• Use cosine learning rate schedule (gradual decay)
• Always include a warmup phase (5% of total steps)

LoRA rank: capacity vs efficiency tradeoff

Rank 4:    Very lightweight, good for simple tasks (classification)
Rank 16:   Good default for most tasks
Rank 32:   More capacity, needed for complex generation
Rank 64:   Near full fine-tune expressiveness
Rank 128:  Rarely needed, diminishing returns

Practical tips

• Start small, scale up: Begin with rank=16, lr=2e-4, 3 epochs
• Watch validation loss: If it starts increasing, you are overfitting
• Effective batch size matters: batch_size x gradient_accumulation_steps x num_gpus
• Gradient checkpointing: Trade compute for memory -- enable it if running out of VRAM
• bf16 > fp16: bfloat16 is more numerically stable, use it when hardware supports it

Slide 22 of 30Open this slide

Evaluating Fine-Tuned Models

Two types of evaluation

1. Benchmark evals (general capability)

Check that your fine-tuned model has not lost general abilities:

• MMLU: General knowledge across 57 subjects
• HumanEval: Code generation
• GSM8K: Grade-school math reasoning
• HellaSwag: Commonsense reasoning

Run these with lm-evaluation-harness:

lm_eval --model hf \
    --model_args pretrained=./my-finetuned-model \
    --tasks mmlu,hellaswag,gsm8k \
    --batch_size 8

2. Task-specific evals (your actual use case)

These matter far more than benchmarks. Design evaluations that mirror production usage.

# Example: evaluating a JSON extraction model
def evaluate_extraction(model, test_set):
    metrics = {"exact_match": 0, "valid_json": 0, "field_accuracy": {}}

    for example in test_set:
        prediction = generate(model, example["input"])

        # Can it produce valid JSON?
        try:
            parsed = json.loads(prediction)
            metrics["valid_json"] += 1
        except json.JSONDecodeError:
            continue

        # Does it match expected output?
        if parsed == example["expected"]:
            metrics["exact_match"] += 1

        # Field-level accuracy
        for field in example["expected"]:
            if field not in metrics["field_accuracy"]:
                metrics["field_accuracy"][field] = {"correct": 0, "total": 0}
            metrics["field_accuracy"][field]["total"] += 1
            if parsed.get(field) == example["expected"][field]:
                metrics["field_accuracy"][field]["correct"] += 1

    return metrics

The evaluation matrix

Always compare against baselines

1. Base model + zero-shot prompt
2. Base model + few-shot prompt
3. Teacher model (upper bound)
4. Your fine-tuned model

If your fine-tuned model does not beat baseline #2, something is wrong.

Sources

• lm-evaluation-harness (EleutherAI)

Slide 23 of 30Open this slide

Common Failures in Fine-Tuning

1. Catastrophic forgetting

Symptom: Model excels at your task but cannot do anything else. Ask it a general question and it responds with task-specific output.

Cause: Too many epochs, learning rate too high, or dataset too narrow.

Fix:

• Reduce to 1-3 epochs
• Lower learning rate (try 1e-4 instead of 2e-4)
• Mix in 10-20% general instruction data (e.g., Alpaca or OpenAssistant)
• Use LoRA instead of full fine-tuning

2. Overfitting

Symptom: Training loss drops to near zero but validation loss increases. Model memorizes training examples verbatim.

Cause: Too few unique examples, too many epochs, rank too high.

Fix:

• Add more diverse training data
• Reduce epochs (watch validation loss curve)
• Increase LoRA dropout to 0.1
• Reduce LoRA rank
• Use early stopping: load_best_model_at_end=True

3. Format collapse

Symptom: Model ignores output format requirements. Asked for JSON, responds in prose. Asked for bullets, gives paragraphs.

Cause: Insufficient format-specific examples, or format examples not diverse enough.

Fix:

• Ensure every training example demonstrates the correct format
• Add explicit format instructions in the system prompt
• Include "negative" examples: malformed format -> correct format
• Increase the proportion of format-heavy examples

4. Reward hacking (DPO/RLHF)

Symptom: Model produces verbose, sycophantic responses that score well on preference metrics but are not actually helpful.

Cause: Preference data is biased toward longer or more agreeable responses.

Fix:

• Include preference pairs where concise answers are preferred
• Add length-controlled pairs (short preferred over long for simple questions)
• Lower the DPO beta to constrain divergence from the base model

5. Repetition loops

Symptom: Model gets stuck repeating phrases or sentences.

Cause: Repetitive training data, or training on outputs with repetition.

Fix:

• Deduplicate training data aggressively
• Use repetition penalty at inference time: repetition_penalty=1.1
• Filter training data for examples containing repetition

Debugging checklist

[ ] Validation loss is decreasing (not just training loss)
[ ] Model can still answer general questions
[ ] Output format matches expectations on >95% of test cases
[ ] No verbatim training examples in outputs (memorization check)
[ ] Performance beats base model + prompting baseline

Sources

• Alpaca: A Strong, Replicable Instruction-Following Model (Taori et al., 2023)
• OpenAssistant (LAION-AI)
• LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)
• DPO: Direct Preference Optimization (Rafailov et al., 2023)

Slide 24 of 30Open this slide

End-to-End Distillation Pipeline

Phase details with cost estimates

Total cost for a typical project

Teacher API calls (10K examples):     $100-300
Quality filtering (LLM judge):         $20-50
GPU compute (QLoRA training):           $5-20
Evaluation compute:                     $5-10
                                      ----------
Total:                                 $130-380

vs. running the teacher model in production:
10K calls/day x $0.01/call x 30 days = $3,000/month

ROI: Pays for itself in ~4 days

Slide 25 of 30Open this slide

Iterative Distillation: Refine Based on Student Failures

The key insight

Your first distillation pass will not be perfect. The student will fail on edge cases, rare categories, and inputs that differ from the training distribution. Use these failures to improve.

The iterative loop

Mining failures for targeted data generation

# After evaluation, collect failure cases
failures = [ex for ex in test_results if ex["correct"] == False]

# Categorize failures
failure_categories = {}
for f in failures:
    category = classify_failure(f)  # e.g., "long_input", "ambiguous", "multi_label"
    failure_categories.setdefault(category, []).append(f)

# Generate targeted synthetic data for each failure category
for category, examples in failure_categories.items():
    prompt = f"""Generate 50 training examples similar to these failure cases.
    The model struggled with: {category}
    Example failures:
    {json.dumps(examples[:5], indent=2)}

    Generate diverse examples that cover this failure mode."""

    new_data = teacher.generate(prompt)

Curriculum learning approach

• Round 1: Easy, clear-cut examples (build basic competency)
• Round 2: Medium difficulty, add ambiguity and edge cases
• Round 3: Hard examples, adversarial inputs, rare categories
• Round 4: Mix of all difficulties (reinforce and stabilize)

When to stop iterating

• Task-specific eval accuracy plateaus (less than 1% improvement per round)
• You have hit the student model's capacity ceiling
• Diminishing returns on teacher API spending
• The student matches or exceeds your quality threshold

Practical tip: the "hard example" multiplier

If the student fails on 5% of cases, generate 10x more training examples for those cases relative to easy cases. This re-balances the training distribution toward the model's weak spots.

Slide 26 of 30Open this slide

Model Merging: Combining Multiple Fine-Tunes

The idea

You have multiple LoRA adapters (or full fine-tunes) specialized for different tasks. Merge them into a single model that can do everything.

Merging methods

TIES (Trim, Elect Sign, Merge)

• Trims small weight changes (noise)
• Resolves sign conflicts between models
• Merges remaining weights by averaging
• Best for: Merging 3+ models with overlapping capabilities

DARE (Drop And REscale)

• Randomly drops a fraction of weight changes
• Rescales remaining weights to compensate
• Reduces interference between models
• Best for: Merging models with conflicting specializations

SLERP (Spherical Linear Interpolation)

• Interpolates between two models along a spherical path
• Preserves the magnitude of weight vectors
• Single interpolation parameter t controls the blend (0.0 to 1.0)
• Best for: Smooth blending of two models

Using mergekit

pip install mergekit

# merge_config.yml
models:
  - model: ./base-model
    parameters:
      weight: 1.0
  - model: ./lora-customer-support
    parameters:
      weight: 0.5
  - model: ./lora-code-generation
    parameters:
      weight: 0.3

merge_method: ties
parameters:
  density: 0.5      # TIES: fraction of weights to keep
  normalize: true
dtype: bfloat16

mergekit-yaml merge_config.yml ./merged-output --cuda

When to merge vs. serve separately

Warning: merging is not guaranteed to work

• Always evaluate the merged model on all tasks
• Quality often drops 5-15% compared to individual adapters
• Iterative merging with evaluation is essential

Sources

• TIES-Merging: Resolving Interference When Merging Models (Yadav et al., 2023)
• Language Models are Super Mario / DARE (Yu et al., 2023)
• mergekit (Arcee AI)

Slide 27 of 30Open this slide

Serving LoRA Adapters at Runtime

The multi-adapter pattern

Keep one base model in memory. Swap LoRA adapters per request for different tasks.

Memory: Base model (4.5 GB quantized) + each adapter (~20 MB) = serve 50 tasks in 5.5 GB

vLLM with LoRA adapters

from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest

# Load base model once
llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    enable_lora=True,
    max_lora_rank=64,
    max_loras=4,  # Max concurrent adapters in memory
)

# Serve different tasks by swapping adapters
support_request = LoRARequest("support", 1, "./lora-support")
code_request = LoRARequest("code", 2, "./lora-code")

# Route based on task
output = llm.generate(
    "Help me with my billing issue",
    SamplingParams(temperature=0.3, max_tokens=512),
    lora_request=support_request,
)

Ollama with custom models

# Create a Modelfile
cat > Modelfile <<EOF
FROM llama3.1:8b
ADAPTER ./my-lora-adapter.gguf
SYSTEM "You are a customer support agent for Acme Corp."
PARAMETER temperature 0.3
EOF

ollama create acme-support -f Modelfile
ollama run acme-support "I need to cancel my subscription"

Production serving checklist

• Adapter hot-loading (no server restart to add new adapters)
• Request routing (which adapter for which request)
• Adapter version management (A/B test new adapters)
• Fallback to base model if adapter fails
• Latency monitoring per adapter
• Adapter caching strategy (LRU eviction for memory management)

Adapter versioning strategy

/adapters/
  /customer-support/
    v1.0.0/  (initial training)
    v1.1.0/  (added edge cases)
    v2.0.0/  (retrained on new product line)
  /code-review/
    v1.0.0/

Tag adapters like software releases. Roll back if quality regresses.

Sources

• vLLM: High-Throughput LLM Serving
• Ollama
• LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)

Slide 28 of 30Open this slide

Key Takeaways

Distillation is a Superpower

Use frontier models as teachers to generate high-quality synthetic training data. The student does not need to match the teacher everywhere -- just on your specific task. This is the most cost-effective way to deploy AI at scale.

LoRA Makes Fine-Tuning Accessible

You do not need a cluster of A100s. QLoRA lets you fine-tune an 8B model on a single consumer GPU in under an hour. The barrier to fine-tuning is now data quality, not compute.

Data Quality > Data Quantity

1,000 excellent examples beat 50,000 mediocre ones. Invest in filtering, deduplication, diversity, and human spot-checks. Your model is only as good as its training data.

Evaluate Like You Deploy

Benchmarks tell you if the model is broken. Task-specific evals tell you if it is useful. Always compare against the baselines: base model + prompting, and the teacher model itself.

Iterate Relentlessly

First-pass distillation rarely achieves target quality. Mine failures, generate targeted examples, retrain, and repeat. Budget for 2-4 rounds of refinement.

Ship Adapters, Not Models

Serve a single base model with swappable LoRA adapters. Each adapter is 20 MB, version-controlled, and independently updatable. This is the deployment pattern for multi-task production systems.

Slide 29 of 30Open this slide

Key Papers and Resources

Essential reading

Practical resources

• Hugging Face PEFT: Comprehensive guide to LoRA, QLoRA, and other adapters
• TRL (Transformer Reinforcement Learning): SFT, DPO, PPO trainers
• Axolotl: YAML-driven fine-tuning with many model support
• Unsloth: Optimized training kernels for fast LoRA fine-tuning
• mergekit: Model merging toolkit (TIES, DARE, SLERP)
• lm-evaluation-harness: Standardized benchmark evaluation
• vLLM: High-performance inference with LoRA adapter support

Community hubs

• Hugging Face Hub: Model hosting, datasets, and spaces for demos
• Open LLM Leaderboard: Compare model performance across benchmarks
• r/LocalLLaMA: Community discussion on fine-tuning and deployment
• Alignment Forum: Research discussion on preference learning and safety

Slide 30 of 30Open this slide

Week 13 Wrap-Up

You now know how to:

1. Generate synthetic training data using teacher models
2. Fine-tune with LoRA and QLoRA on a single GPU
3. Evaluate and iterate on distilled models
4. Deploy adapters in production

Next week: Safety & Red-Teaming

Test prompt injection and agent hijacking, apply layered controls, and measure remaining failures.

"The best model is not the biggest one -- it is the one that ships."