Private AI for business

What "private AI" actually means

The phrase gets used for three quite different things, and conflating them is the source of most confusion in vendor conversations.

A cloud AI product with a privacy promise. Your data goes to a vendor who contractually agrees not to train on it. This is what most products labelled "enterprise AI" are. The promise is real; the disclosure still happens.

A dedicated cloud tenant. Your own isolated environment with a cloud provider. No shared inference, nothing co-mingled, but still somebody else's infrastructure.

Genuinely self-hosted. Models running on hardware you control, inside your network. No outbound call to any model provider.

This guide is about the third, with notes on the second. The distinction matters because the obligations that force you into self-hosting are usually about disclosure rather than about training, and only the third option removes disclosure.

When self-hosting is the right call

Five conditions. You probably need one of them to justify it, and more than one makes it obvious:

1. A contract or regulation forbids third-party disclosure. Privilege, an NDA naming permitted processors, a data processing restriction. The binding question is not "will they train on it" but "may this information reach them at all".
2. The documents are your competitive position. Formulations, tooling specifications, proprietary methods. A vendor's retention policy is excellent today and subject to revision.
3. Per-seat cost is growing faster than value. Fixed infrastructure and rapid hiring favour self-hosting; small stable teams do not.
4. You need an audit trail you control, with your own retention rules, because someone will eventually ask to see it.
5. Offline or air-gapped operation. No further argument required.

If none of those apply, a commercial subscription is very likely the better purchase and we would tell you so. We set the comparison out in full in Private AI vs ChatGPT Enterprise.

The four components of a working system

People think of private AI as "a model". The model is the component you will replace most often and worry about least.

• The model runtime. Serves an open-weight model on your hardware. Replaceable — this is deliberately the least sticky part of the stack.
• Retrieval. Your documents, chunked, embedded into a vector index, searched at query time. This is where deployments succeed or fail, and it gets far less attention than it deserves.
• The interface. What staff actually use. An excellent model behind an interface nobody opens produces zero value, and this is the most common reason a pilot quietly dies.
• Access control and audit. Who can ask what, which documents they can reach, and a log of every query and retrieval.

Retrieval versus fine-tuning — the decision people get wrong

Almost everyone who asks for a model "trained on our data" wants retrieval, not fine-tuning. The distinction is worth being precise about because getting it wrong is expensive.

Retrieval looks up relevant passages from your documents and gives them to the model as context. Updating means re-indexing a file. Answers can cite their source.

Fine-tuning adjusts the model's weights to shift its style or behaviour. It does not reliably teach facts, it cannot cite anything, and updating means retraining.

If you want the system to know what your policy says, that is retrieval. If you want it to write in a particular house style or follow an unusual output format consistently, that is fine-tuning. Most business requirements are the first, and we say so in the custom-trained models service — which begins by measuring whether retrieval already solves it.

What hardware you actually need

The honest answer is that it depends on model size, how many people use it at once, and how fast you need answers. The useful answer is the shape of the decision:

• Small team, document retrieval. A mid-size open-weight model on a single workstation-class GPU is frequently sufficient. Plenty of businesses already own something adequate.
• Department scale, concurrent use. A proper GPU server. This is where real capital expenditure starts.
• Company-wide. Multiple GPUs or a private cloud tenant, and a genuine decision about capital versus operating cost.
• CPU-only. Technically possible, usually too slow for interactive use. Fine for overnight batch work.

Worth saying: We will not quote you hardware before seeing what you are trying to do. If you have capable machines sitting idle, the honest recommendation is usually to start with them and buy later, once real usage has told you what you need.

What it costs — and how to work it out yourself

We are not going to print a made-up table of monthly figures. Hardware prices move, model efficiency moves, and a fabricated comparison is exactly what should make you distrust a vendor guide.

What is durable is the shape of the two cost curves, and you can calculate your own crossover in an afternoon:

1. Count your seats and what you pay per seat per month for commercial AI today. Multiply out for a year. Then do it again at the headcount you expect in two years.
2. Price the hardware for the deployment shape above, or get a private cloud tenant quote. Add deployment and the first year of maintenance.
3. Compare the totals over three years. Per-seat is a line that rises with hiring; self-hosted is a step followed by a shallow slope.
4. Add the value of the work you currently cannot do at all because the data is not allowed near a public model. For regulated businesses this is frequently the largest term in the calculation and it is the one nobody puts in the spreadsheet.

For token-level economics on the commercial side, we publish what we actually pay per engine at AI engine pricing and pass it through at cost. An AI cost analysis runs this against your real numbers, and does sometimes conclude that you should keep paying per seat.

How deployment actually runs

1. Assessment. What staff need, which documents matter, what your network and identity setup looks like. Most of the consequential decisions are made here.
2. Model and hardware selection. Sized to the work rather than to a leaderboard.
3. Knowledge ingestion. Documents chunked and indexed, with retrieval scoped so teams only reach what they should.
4. Access control and audit. Permissions mapped to existing groups; logging and retention configured.
5. Pilot with one real team. Real work, daily feedback. Retrieval scoping is always slightly wrong at first and this is where it gets fixed.
6. Expand and maintain. Further groups as the pilot settles.

Why these projects fail

In roughly this order of frequency:

• Nobody owned it. The single biggest cause. A system with no internal owner decays regardless of who maintains it.
• The documents were a mess. Retrieval over disorganised, contradictory content produces confidently wrong answers faster than a human would. Fix the content first.
• The interface was an afterthought. Staff will not use something awkward, however good the model is.
• Retrieval scope was too broad. Everything indexed into one pool, so answers pull from irrelevant or restricted documents and trust collapses after the first bad answer.
• It was sold as replacing judgement. Expectations set there are never met, and the disappointment kills adoption even when the system works.
• No measurement. Without a before state, nobody can say whether it helped, and the budget goes elsewhere next year.