What it actually takes to run a full AI assistant on hardware you control, what it costs to serve it to many users, and the conditions under which data sovereignty pays for itself.
Why it is worth discussing, and on what condition.
Letting OpenAI or Anthropic process your data means sending it, in clear text, to third-party infrastructure. For many applications that trade-off is acceptable. For sensitive data — health, legal, industrial, personal records — it is a real limit.
Data sovereignty — AI running on hardware you control — is the structural answer. It is a long-term bet: it only makes sense with a client base large enough to justify the fixed cost of the hardware. This document measures exactly where that threshold lies.
A real assistant is not a one-shot question to a model.
A useful AI assistant does not answer an isolated prompt: it carries operating instructions, project memory, and the definition of the tools it can use. In our real case, every single message carries a context of ~40,000 tokens before the user's question even begins.
On a local model, processing 40,000 tokens of context costs time on every turn. The perceived symptom: the chat feels slow, every message makes you wait. The cause is not the model "thinking": it is the context being reprocessed from scratch each time.
Cache reuse is everything. Three obstacles, one fix.
The key to making a local assistant responsive is cache reuse: the stable context (the instructions, the tools) should be processed once and reused on later messages. In theory this cuts the wait from 12 seconds to fractions of a second. In practice, three obstacles prevented it.
Some modern models use a hybrid attention that makes them efficient on long context but, as a side effect, prevents cache reuse: every turn starts over. A classic-attention model does not have this limit.
Only one "hot" model fits on a single machine at a time. Any request asking for a different one — even a routine automated health check — evicts the model and loads another, wiping the cache. A check every few seconds was enough to ruin everything.
The assistant framework injected a small code at the head of the context that changed on every request. It serves the originating cloud provider's cache; redirected to the local model it became the opposite — a cache saboteur, breaking reuse from the very first character.
The fix is a normalization layer in front of the inference engine that: always serves a single model (no contention), neutralizes automated health checks (no evictions), and stabilizes the context (no sabotage). It is independent of the model and of the client.
On the same hardware, intelligence and speed are the same lever.
"Mixture-of-experts" models activate only a small fraction of their parameters for each word: they are very fast but reason less — they get facts wrong and drift into style. "Dense" models use all their parameters: sharper, but slower. The larger the model, the more hardware it takes to keep it alive.
| Profile | Intelligence | Speed | Verdict |
|---|---|---|---|
| Experts, few active | Low | very high | turbo, but imprecise |
| Dense ~70B | High | low* | serious |
| Top-tier cloud | Very high | high | not local |
*on entry-level hardware. With the right tier a dense 70B model becomes fully responsive.
The unit of cost is throughput, not the user.
A GPU machine is a fixed budget of work per second, shared across all users. The 40,000-token context is the dominant factor: shared-prefix caching — many users of the same service sharing the same base context, processed once and reused by all — is what decides whether one machine serves five users or fifty.
| Hardware | Fee/month | Break-even | Model | Users* |
|---|---|---|---|---|
| Entry (48 GB) | ~€350 | ~2,500 req/d | lightweight | ~80 |
| Mid (96 GB) | ~€1,000 | ~7,000 req/d | serious 70B | ~230 |
| High (160 GB) | ~€2,500 | ~17,000 req/d | 70B+ / 123B | ~550 |
*estimate at ~30 requests per user per day. The break-even is high because the cloud's per-token cost is very low.
Not all on-premise is sovereignty.
A distinction often lost: renting a GPU machine does not grant data sovereignty. The data is still processed on someone else's hardware. It takes the large AI providers out of the loop, but it does not guarantee physical control.
True sovereignty — data that never leaves hardware under your control — requires your own hardware, on-premise. And there you accept a quality ceiling in exchange for budget: the model will be whatever your machine can hold, not the most powerful on the market.
Maximum quality, minimal cost. No sovereignty.
Outside the big providers. Partial sovereignty.
Real sovereignty. Quality ceiling set by budget.
A staged path, tied to the number of clients.