Private AI guide
Should you fine-tune a model or use retrieval?
Nearly everyone who asks us to train a model on their data wants retrieval. The distinction is worth ten minutes because getting it wrong costs months.
What each one actually does
Retrieval leaves the model alone. At query time it searches your documents, pulls the relevant passages, and hands them to the model as context. The model reasons over text it was given.
Fine-tuning changes the model's weights using example inputs and outputs. It shifts behaviour, style and format. It is poor at installing facts, and it cannot cite anything.
Why retrieval wins for most business cases
- Updates are trivial. A policy changes, you re-index one file. A fine-tune would need retraining.
- Answers cite sources. Critical for anything a person has to act on — nobody will use a tolerance figure or a contract clause without knowing which document it came from.
- Access control works. Retrieval can be scoped per team. A fine-tuned model has absorbed everything it was trained on and cannot un-know it for a particular user.
- It is far cheaper, because there is no training data to assemble.
- Removal is possible. Delete a document from the index and it is gone. Removing something from fine-tuned weights means retraining.
When fine-tuning is genuinely right
- A consistent output format that prompting keeps drifting away from.
- A specific voice or house style across large volumes of text.
- A narrow classification task with plenty of labelled examples, where a small fine-tuned model beats a large general one on both cost and latency.
- Latency or cost pressure where a small specialised model replaces a large general one at volume.
Worth saying: Notice what is absent from that list: knowing your documents. That is always retrieval.
How to decide in one question
Ask: am I trying to change what it knows, or how it behaves?
What it knows is retrieval. How it behaves is fine-tuning. If the answer is both, do retrieval first — it is cheaper, faster and frequently sufficient, and you will understand the remaining behaviour gap much better afterwards.
Our custom-trained models service starts by measuring a retrieval baseline for exactly this reason, and a good share of those engagements stop there.
Questions people actually ask
Before you call
Can we do both?
Yes, and for a narrow high-volume task it is sometimes the right answer — a fine-tuned model for the behaviour, retrieval for the facts. Start with retrieval.
How much data does fine-tuning need?
Consistent, labelled examples in at least the high hundreds for most tasks. Most businesses discover they do not have them yet, and assembling them is the actual project.
Does retrieval work on scanned documents?
Once they are text. That is what Felican IDP is for.
Still have the question?
Ask us directly. We answer these on calls all day.
