Ask a vendor what on-premise AI costs and you get a hardware quote. Ask somebody who runs it and you get a longer answer, most of which is not about hardware.
We deploy and operate these systems on customer machines. The quote is the part everyone gets right.
The GPU is the half you can see
A serious inference box, meaning one that runs a capable open-weight model at a speed people will actually tolerate, is a real capital purchase and there’s no clever way around it. It is also roughly half of what the system costs to have.
Power and cooling is the first surprise. A machine drawing over a kilowatt under sustained load is a space heater with opinions, and the office closet somebody earmarked for it usually can’t take the heat or the circuit. That’s a facilities conversation, and it’s a much better conversation to have before the pallet arrives.
Redundancy is the second. One box means every driver update is a planned outage. Whether that matters depends entirely on the workload, and it’s far cheaper to decide while you’re still drawing boxes than after someone asks why the assistant is unavailable on a Tuesday morning.
The third is people. Somebody has to update drivers, roll models forward, and notice when throughput quietly halves because a kernel update changed a default. That is either your team’s time or a support agreement with us. It is never zero, and it’s the line most often missing from the spreadsheet.
Quantization is where the money actually is
Full-precision weights are what the benchmarks advertise. Quantized weights are what almost every on-premise deployment actually runs, because dropping the precision cuts memory requirements sharply for a quality cost that, on most real tasks, nobody can find.
This is the single biggest lever on entry cost. The same budget buys a substantially more capable model at four or eight bits than at sixteen, and for extraction, classification and internal assistant work the lower-precision version is usually indistinguishable in daily use.
Test it on your own documents. Leaderboard scores are measured on tasks that aren’t yours, and the gap between a model that benchmarks well and a model that reads your invoices well is wide enough to make the decision for you.
Buy less than you think, deliberately
The failure mode we see isn’t underspending. It’s overbuying.
A cluster gets approved, racked and burned in, and then the workload turns out to need a fraction of it. The money is spent, the utilization graph is embarrassing, and the person who championed the project now has a political reason to keep it busy with work that never needed a GPU.
So we prove the workload on one node first. A single box that turns out to be the wrong answer becomes a build server, a render node, or somebody’s very good workstation. A cluster that turns out to be the wrong answer stays a cluster. Scale when the queue depth says to, not when the roadmap slide does.
When your own metal wins
Cloud APIs win on convenience. They lose on two axes, and only two.
The first is volume. Push enough tokens through a hosted model every day and the per-token bill eventually crosses the cost of owning the machine that would have done it. That crossover is arithmetic rather than ideology, and for plenty of companies it never arrives. We say so when it doesn’t.
The second is the data boundary, and it isn’t a cost question at all. A data processing agreement is a promise about what someone else will do with your data. A machine in your own rack is a statement about where your data is. Those are different kinds of thing, and organizations handling security findings, unpublished research or medical records tend to already know the difference. For them the spreadsheet was never the deciding document.
If you’re weighing the two, we do the assessment as a scoped engagement: your workload, your constraints, a written recommendation with the numbers in it. Details are on the On-Premise AI page, or just tell us what you are running.