Fine-Tuning LLMs: When and How

The trap everyone falls into

Your model keeps answering in the wrong tone. It ignores your JSON schema. It writes like a chatbot when you need a terse internal tool. So someone says the magic words: "let's just fine-tune it." Three weeks later you have a labeled dataset, a training bill, a checkpoint that is already stale, and the model still hallucinates yesterday's pricing, because fine-tuning was never the tool for that problem.

Fine-tuning is powerful and badly misunderstood. The single most useful thing you can learn about it is what it cannot do. Get that right and the rest is mechanical.

One principle, one analogy

Fine-tuning teaches a model how to behave. It does not teach it what is true today.

Think of a base model as a brilliant new graduate. They reason well, write well, and know an enormous amount about the world up to the day they stopped reading. Fine-tuning is sending them to a finishing school for tone and craft, teaching them your house style, your output format, your domain's voice. What it is *not* is handing them this morning's newspaper. For fresh facts you put the newspaper on their desk at question time. That desk is your prompt, and the newspaper is retrieval.

The fine-tuning workflow

When fine-tuning *is* the right call, the path is the same every time. Curate a clean dataset, format it the way the trainer expects, train a small adapter (not the whole model), evaluate honestly, deploy, and then watch it in production. The diagram below is the loop; the steps after it walk one pass through it.

1. 1

   Curate the dataset

   Collect a few hundred to a few thousand high-quality input/output pairs that demonstrate exactly the behavior you want. Quality beats quantity by a mile, 500 clean examples outperform 5,000 noisy ones.

2. 2

   Format to the chat schema

   Convert each pair into the JSONL message format your trainer expects (system / user / assistant turns). Consistency here is non-negotiable; one malformed field can skew the whole run.

3. 3

   Train a LoRA adapter

   Freeze the base model and train a small set of low-rank adapter weights. Minutes to a couple of hours on a single GPU, not a data-center job.

4. 4

   Evaluate against a held-out set

   Score on examples the model never saw during training, plus an LLM-judge or human pass for quality. If it does not beat your prompt-only baseline, stop, do not deploy.

5. 5

   Deploy the adapter

   Serve the small adapter on top of the base model. You can keep several adapters and swap them per use case without re-hosting the base.

6. 6

   Monitor and feed back

   Watch for regressions and distribution drift in production, harvest fresh examples, and feed them into the next curation round.

Prompting vs RAG vs fine-tuning

These three are not competitors, they solve different problems and stack happily. The mistake is reaching for the most expensive one first. Try prompting. Then try RAG. Only then consider fine-tuning. Most teams never need the third box.

What the data actually looks like

The dataset is the project. Each line is one self-contained example in the chat schema, a system message setting the role, the user input, and the assistant output you wish the model had produced. Below is a tiny slice of a support-triage dataset teaching the model to always reply with a fixed JSON structure.

{"messages": [{"role": "system", "content": "You are a support triage bot. Reply only with JSON."}, {"role": "user", "content": "My invoice is wrong and I want a refund."}, {"role": "assistant", "content": "{\"category\": \"billing\", \"priority\": \"high\", \"needs_human\": true}"}]}
{"messages": [{"role": "system", "content": "You are a support triage bot. Reply only with JSON."}, {"role": "user", "content": "How do I reset my password?"}, {"role": "assistant", "content": "{\"category\": \"account\", \"priority\": \"low\", \"needs_human\": false}"}]}
{"messages": [{"role": "system", "content": "You are a support triage bot. Reply only with JSON."}, {"role": "user", "content": "The app crashes every time I open a report."}, {"role": "assistant", "content": "{\"category\": \"bug\", \"priority\": \"high\", \"needs_human\": true}"}]}

Notice what these examples teach: format and routing behavior, not facts. None of them depend on information that changes day to day. That is the signature of a good fine-tuning dataset.

Training itself is mostly configuration. A LoRA run reads like this, a base model, a handful of hyperparameters, and the rank that controls how big your adapter is.

base_model: meta-llama/Llama-3.1-8B-Instruct
dataset: ./train.jsonl

lora:
  r: 16              # adapter rank, higher = more capacity, more params
  alpha: 32          # scaling factor, usually 2x r
  dropout: 0.05
  target_modules:    # which layers get adapters
    - q_proj
    - v_proj

training:
  epochs: 3          # too many = memorization / overfit
  learning_rate: 2.0e-4
  batch_size: 8
  eval_split: 0.1    # hold out 10%, never train on your eval set

Evaluating the result

A fine-tune that you did not evaluate is a liability, not an asset. The non-negotiable rule: always compare against your prompt-only baseline on data the model never trained on. If the fine-tune does not clearly beat the baseline, you just spent weeks to ship a regression risk.

• Hold out a test set early. Split before you train and never let those examples leak into training.
• Beat the baseline or stop. Run the same eval against plain prompting and against RAG. The fine-tune has to win to justify itself.
• Measure what you actually fine-tuned for. Format compliance, tone match, schema validity, task-specific metrics, not a generic benchmark score.
• Watch for regressions. A model tuned hard on one task often gets *worse* at everything else. Keep a small general-capability check in the loop.
• Use an LLM judge for the fuzzy stuff, but spot-check the judge against human ratings so you trust it.

For the full discipline of measuring LLM systems, judges, regression suites, and offline-vs-online evals, see Evaluating LLM Applications.

Common mistakes that cost weeks

1. Fine-tuning for fresh facts. Weights freeze at training time. If the answer changes daily, you want RAG, not a fine-tune. This is the number-one wasted project.
2. Too little or dirty data. A few dozen inconsistent examples teach the model noise. Curate for quality; 500 clean pairs beat 5,000 messy ones.
3. No evaluation. Shipping a checkpoint because the loss curve looked nice, with no held-out comparison to the baseline. You have no idea if it is better.
4. Skipping prompting and RAG first. Reaching straight for the most expensive, slowest, hardest-to-change tool when a one-line prompt edit or a retrieval step would have solved it.
5. Overfitting on epochs. Training too long memorizes your examples and destroys generalization. Watch eval loss, not just training loss.
6. One model to rule them all. Cramming many unrelated tasks into one fine-tune. Prefer small, focused adapters you can swap per use case.

Takeaways

Where to go next

Fine-tuning sits near the top of the AI engineering stack, reach for it after you have squeezed prompting and retrieval. Build the layers underneath it first.

• Solve the freshness problem the right way: RAG Architecture Explained.
• Build the eval discipline before you train anything: Evaluating LLM Applications.
• See where fine-tuning fits in the broader role: the AI Engineer career path.