TL;DR
- Fine-tuning wins on cost at volume, latency, rigid output formats and data residency. It rarely wins on general knowledge or taste.
- Below roughly 50,000 requests per month on a single task, operational overhead usually outweighs the saving.
- Measure cost per successful task and an evaluation delta on a frozen set. A demo is not a measurement.
- OpenAI's self-serve fine-tuning wind-down makes provider lock-in a real variable in the decision.
The short answer
Fine-tuning is a structural intervention, not a quality upgrade. It changes the model's weights so a behaviour becomes cheap and repeatable. Prompting changes the instructions at request time, which is flexible but pays a token cost on every call and depends on the model following a long instruction every time.
That distinction explains most of the confusion in the fine-tuning debate. Teams often ask "will fine-tuning make our AI better?" The honest answer is: better at what, measured how, and at what volume? Fine-tuning is the right buy for four specific problems. For everything else, it is an expensive detour.
A second distinction matters just as much: knowledge versus behaviour. If the model needs to know a new policy that changed last week, use retrieval. If the model needs to stop producing a malformed JSON field, fine-tuning can teach that behaviour once instead of repeating the schema in every prompt.
What does fine-tuning actually change?
A model's weights encode a distribution of behaviour. A prompt is a temporary instruction inside that distribution. When the instruction is short and the model already knows how to follow it, prompting is the cheapest possible intervention.
Fine-tuning moves the behaviour into the weights. The cost of the instruction disappears, the model needs less context, and the output becomes more consistent. The price is that you now own a model artifact: a dataset, a training run, an evaluation, a serving configuration and a maintenance plan.
That trade is worth it when the problem is structural. It is rarely worth it when the problem is "the model is not smart enough" in general. A fine-tune does not add reasoning capability that the base model does not have. It makes a narrower behaviour reliable.
The four cases where fine-tuning wins
| Problem | Why prompting struggles | What fine-tuning changes | What to measure |
|---|---|---|---|
| Cost at volume | Every request repeats the same examples and schema, and you pay for them every time | The behaviour is learned once; the prompt gets shorter or disappears | Cost per successful task at your real volume |
| Latency | A large model plus a long prompt adds queue and prefill time | A smaller model with a shorter prompt can run closer to the application | p50 and p95 latency, not one warm request |
| Rigid output formats | The model's prior disagrees with your schema, so instructions need constant reinforcement | The format becomes the path of least resistance | Schema compliance on a frozen test set |
| Data residency | Inference cannot leave a named environment | Open weights can run on infrastructure you control | Quality and latency on the target hardware |
Notice what is missing: "better answers" in general, "more creative", "smarter". Those are not fine-tuning problems. They are model choice, retrieval, product design or evaluation problems.
The strongest case is usually a combination. A document extraction task might need a smaller model for latency and cost, a fine-tune for format compliance, and a retrieval layer for the policy behind the form. Each intervention should be justified by a different measurement.
The volume threshold
Fine-tuning has a fixed cost: dataset work, training, evaluation, serving and the ongoing cost of maintaining a model artifact. The per-request saving has to amortise that fixed cost.
The published estimate that is most useful in practice is roughly 50,000 requests per month on a single task. Below that, the operational overhead often dominates. Above it, the arithmetic starts to work, especially when prompts are long and the task is narrow.
Treat that number as a starting point, not a law. The real crossover depends on:
- input and output token counts;
- the price of the frontier model you replace;
- the size of the model you can serve;
- GPU cost and utilisation;
- how much human review the current system needs.
The measurement that matters is cost per successful task:
cost per successful task = (token cost + retries + human review + infrastructure) / successful outcomes
Most teams measure cost per token. That is a vendor metric. Cost per successful task is the one that survives a budget review, because it includes the retries and reviews that a cheaper model can quietly multiply.
When should you not fine-tune?
Fine-tuning is the wrong tool in five common situations.
- You have no real usage data. Without a sample of real inputs, there is no honest evaluation set and no way to know whether the fine-tune helped.
- The task changes every week. A prompt can change in minutes. A fine-tune needs a new dataset, a new run and a new evaluation.
- The problem is missing knowledge. A fine-tune can teach a format or a behaviour, but it is a poor way to keep facts current. Retrieval is the right layer for changing knowledge, and it gives you citations.
- You need general reasoning. A small fine-tune does not turn a 3B model into a frontier model. If the task needs broad reasoning, choose a bigger base model and optimise somewhere else.
- You cannot measure the result. If nobody can define a correct answer and build a test set, the project will end with an argument rather than a number.
There is also a provider-specific trap. In May 2026 OpenAI announced it is winding down self-serve fine-tuning: new organisations were cut off, existing active users lose access on 2 July 2026, and no new fine-tuning jobs are accepted after 6 January 2027. The stated reason is that newer base models follow instructions and formats well enough that prompting is often cheaper. Whatever you think of that decision, it is a reminder that a proprietary fine-tune is a dependency, not an asset. Open weights are portable.
How do you prove it worked?
A fine-tuning project without a frozen evaluation set is a demo with a training bill. The protocol is not complicated, but it has to be fixed before training starts.
- Freeze the evaluation set. 30 to 80 real cases, with the expected output and a pass/fail rule. Include common cases, known failures, edge cases and a few adversarial inputs.
- Baseline the current system on that set. The baseline is your current prompt, current model and current retrieval. Not a strawman.
- Train on a separate split. Never evaluate on examples the model saw during training. Leakage produces beautiful numbers and a broken system.
- Compare under identical conditions. Same test set, same decoding settings, same scoring script.
- Report the failure analysis. A single score hides the categories that matter. Show which cases improved, which regressed and why.
- Measure the operational numbers. Cost per successful task, p50 and p95 latency, memory, and the human review rate.
- Hand over the harness. The evaluation is the durable asset. The model will be replaced; the way you decide whether it is better should not be rebuilt every time.
For a worked example of a rigorous protocol, see the MTF-Qwen case study on this site: a fully fine-tuned Qwen3-0.6B, a custom trajectory-aware loss, six baselines and a frozen test set under the FinTSBridge protocol.
The decision in one page
Use this as a checklist before approving a fine-tuning sprint.
- One narrow task, with a definition of a successful outcome.
- Real usage data, and permission to use it.
- A current baseline measured on the same data.
- A volume estimate that makes the fixed cost amortise.
- A frozen evaluation set and a written rubric.
- A clear answer to "what happens when the data changes?"
- A serving plan for the model, including the hardware.
- An exit rule: if the eval delta is below the agreed threshold, stop.
If most of those boxes are empty, the first project is not a fine-tune. It is an evaluation audit, which is cheaper and tells you whether the fine-tune is worth doing at all.
What I do about it
I run fixed-price fine-tuning sprints for narrow, high-volume tasks: dataset design, LoRA or full fine-tuning, quantization, a frozen evaluation set and a before/after benchmark. If the numbers do not justify the spend, the audit says so before the sprint budget is committed.
If you are trying to decide between prompting, retrieval and fine-tuning, start with a workflow audit or talk to me about the model decision.
Questions
Is fine-tuning cheaper than prompting?
Only at volume. Fine-tuning adds dataset work, training runs, evaluation and an operations surface. It becomes cheaper when the same narrow task runs often enough for the per-request saving to cover that overhead, or when latency or data residency has a cost that an API cannot match.
How much training data do I need?
Hundreds to a few thousand high-quality examples, depending on the task. A smaller set of clean, representative examples usually beats a large set of noisy ones. The evaluation set matters more than the training set size, because it tells you whether the fine-tune helped.
Can I fine-tune a proprietary model?
Sometimes, but check the provider's roadmap first. OpenAI announced in May 2026 that it is winding down self-serve fine-tuning, with existing active users losing access on 2 July 2026 and no new jobs after 6 January 2027. Open-weight models avoid that specific risk because you own the weights.
Does fine-tuning stop hallucinations?
No. Fine-tuning changes behaviour, format and task-specific skill. It does not guarantee that a claim is grounded in a source. Grounding needs retrieval, citations and an evaluation that checks faithfulness separately from fluency.