Fine-Tuning and Local Models: The Two Questions I Couldn't Answer, Answered.

In my last article, I told the story of blanking on “what is a harness?” during an interview with company X.

What I didn’t say: that wasn’t the only gap.

There were two more.

“What are the main fine-tuning approaches, and when would you use each one?” “How would you pick and deploy a local model in production?”

Both times, the same honest answer: I don’t know.

10 years shipping mobile apps. AI agents running in my stack every single day. And two fundamental questions where all I had was a vague intuition.

So I did the same thing as last time. One week of focused learning. And I came out with a mental map I can hand you in a 10-minute read.

Here it is.

Part 1: the 3 families of fine-tuning

Fine-tuning means taking an already-trained model and specializing it on your data.

Sounds simple. In practice there are three very different approaches, and mixing them up is how you burn budget for nothing.

Family 1: full fine-tuning

You retrain every weight in the model on your data.

It’s the most powerful approach. It’s also by far the most expensive.

Why? Because training isn’t just running the model. You need to store gradients and optimizer states — count 16 to 20 bytes per parameter. For a 7B model, that’s over 120 GB of VRAM. You don’t do that on your laptop. You rent a GPU cluster.

When to use it:

• you want to deeply change what the model is (new domain, a language it barely covers)
• you have tens of thousands of high-quality examples
• you have the GPU budget and a team to run the training

In other words: almost never, if you’re a small company or an early-stage product. This is the tool of labs and large enterprises.

Family 2: LoRA and QLoRA

This is where it gets interesting for the rest of us.

The LoRA idea: instead of touching the model’s billions of parameters, you freeze everything and train small matrices bolted on the side. Less than 1% of the parameters move.

Concrete result: fine-tuning a 7B model becomes doable on a single 24 GB GPU. One RTX 4090. Not a cluster.

QLoRA pushes further: the base model is compressed to 4 bits during training. A 70B model becomes fine-tunable on roughly 48 GB of VRAM. Two consumer cards.

Bonus: the training output is a small “adapter” file of a few hundred MB. You can keep several for the same base model. One adapter per client, per tone, per task.

When to use it:

• you want a consistent style or output format (brand voice, strict JSON, calibrated summaries)
• you have 500 to a few thousand clean examples
• you want a controlled cost: tens of dollars of GPU time, not tens of thousands

This is the default family. When someone says “we’re going to fine-tune,” 90% of the time they mean this — or they should.

Family 3: RLHF and DPO

Third family, and it’s a different sport: preference alignment.

Here you’re not teaching the model knowledge or a format. You’re teaching it to prefer some answers over others. “This response is better than that one.”

• RLHF: the historical method. You train a reward model on human judgments, then optimize the model against it. Powerful, but heavy and unstable.
• DPO: the simplified version that won. No separate reward model. You feed pairs of “preferred answer / rejected answer” directly, and the model learns from the contrast. Same family of results, much simpler pipeline.

When to use it:

• you want to tune behavior: refusals, caution, helpfulness, tone in ambiguous cases
• you have preference pairs (often from real user feedback)
• you’ve already done supervised fine-tuning first — this is a finishing layer, not a starting point

This is how the models you use every day get polished. But for a typical product, you rarely touch it yourself.

The golden rule: RAG before fine-tuning

Here’s the point I should have led with in the interview. It’s worth more than the three families combined.

Most “we need fine-tuning” cases are actually RAG cases.

The question to ask: is your problem a knowledge problem or a behavior problem?

• “The model doesn’t know our products / our docs / our pricing” → knowledge problem → RAG. You fetch the right information at query time and put it in the context. Always up to date, traceable sources, zero retraining.
• “The model doesn’t answer in the right format / tone / structure” → behavior problem → now fine-tuning (LoRA) makes sense.

Fine-tuning is bad at injecting knowledge. Learned facts go stale, the model hallucinates what it half-memorized, and every data update means retraining.

RAG ships in days, debugs by reading what was retrieved, and updates by editing a document.

So the order is: solid prompting → RAG → fine-tuning. In that order. You only move up a level when the previous one plateaus — with measurements to prove it.

Part 2: picking and deploying a local model

Second question I flunked. It breaks down into four decisions.

Decision 1: VRAM sizing

Rule of thumb, worth memorizing:

VRAM needed ≈ parameter count (in billions) × bytes per parameter, plus 20-30% headroom for context.

Concretely:

• FP16 (the usual full precision): 2 bytes per parameter. A 7B ≈ 14 GB. A 70B ≈ 140 GB.
• Q8 (8-bit): 1 byte. A 7B ≈ 7 GB.
• Q4 (4-bit): roughly 0.5-0.6 bytes. A 7B ≈ 4-5 GB. A 70B ≈ 40 GB.

The headroom is for the KV cache: the longer your context and the more users you serve in parallel, the bigger it grows. That’s what surprises you in production — not the weights.

Practical translation:

• a Mac with 16 GB of unified memory runs a 7B-8B at Q4 comfortably
• an RTX 4090 (24 GB) runs a 14B at Q4, even a tight 32B
• a 70B at Q4 needs about 48 GB: two GPUs, or one big unified-memory machine

Decision 2: Ollama or vLLM

Both run models. They don’t play in the same league.

Ollama is the dev tool. One-command install, ollama run and you’re live, built-in model management. Perfect for prototyping, trying models, running a personal assistant. Its limit: it’s designed for one user at a time.

vLLM is the production tool. Its job is throughput. It batches requests continuously and manages KV cache memory precisely (PagedAttention). Under multi-user load, you’re looking at 10-20x more tokens per second served compared to a naive stack. OpenAI-compatible API, so your client code doesn’t change.

The rule is simple:

• dev, prototyping, solo use → Ollama
• production, concurrent users, cost per token that matters → vLLM

Prototyping on Ollama then deploying on vLLM is a perfectly normal path.

Decision 3: quantization

Quantization means compressing the model’s weights by reducing their precision. Q8 = 8 bits, Q4 = 4 bits.

What you gain: VRAM (half, then a quarter), and speed.

What you lose:

• Q8: almost nothing. The loss is imperceptible in practice. If it fits at Q8, take Q8.
• Q4: a slight but real degradation. The model stays good at conversation and summarization, but gets a bit less reliable on fine-grained reasoning, hard code, and complex instruction following.

And one counter-intuitive rule that’s worth gold: a big quantized model almost always beats a small full-precision model at equal VRAM. A 32B at Q4 ≈ 18 GB crushes a 7B at FP16 ≈ 14 GB. Given the choice, take more parameters and quantize.

Decision 4: the test that matters

Last point, and it’s the most important one.

Public benchmarks (MMLU and friends) tell you a model is good in general. They tell you nothing about your case. Models are partly optimized for those benchmarks — that’s the showroom, not the workshop.

What matters is your golden paths: the 20 to 50 real queries that represent what your product must get right. Your real documents, your real output formats, your real edge cases.

The protocol fits in three lines:

1. build your golden-path set once
2. run it through 2-3 candidate models, at the quantizations you’re targeting
3. compare outputs side by side, and measure what’s measurable (format respected, correct fields, unjustified refusals)

One afternoon of work. And at the end, you pick your model on your data, not on a leaderboard.

It’s the exact same reflex as in mobile: I don’t pick a library by its GitHub stars, I test it on my use case.

The full mental map

If I had to redo that interview, here’s my 30-second answer:

Fine-tuning:

• knowledge problem → RAG, no fine-tuning
• style/format problem → LoRA or QLoRA, one GPU is enough
• deep behavior problem → DPO, with preference pairs
• full fine-tuning → almost never, unless big budget and a real foundational need

Local models:

• sizing: parameters × bytes/param (2 at FP16, ~0.5 at Q4) + 20-30% headroom
• Ollama for dev, vLLM for production
• Q8 without hesitation, Q4 with eyes open, big quantized model > small precise model
• final decision on your golden paths, never on public benchmarks

Two “I don’t know”s turned into a mental map. One week each.

AI is still my accelerator, not my replacement. But an accelerator is easier to drive when you understand the engine.

To wrap up

If you’re currently asking yourself “should we fine-tune?” or “should we go local?” for your product, the answer rarely lives in an article. It lives in your data, your budget, and your golden paths.

I help teams make that call, then ship the simple version that works.

If that’s where you are right now, reach out. A 30-minute call often saves three months in the wrong direction.