Infrastructure / Brain Protocol

Scaling the Local LLM Server

A rented two-GPU box runs one capable model today. The question is not which bigger model to buy, but how many brains the same hardware can serve — and what each one costs. The constraint is video memory, not speed; the cost per user collapses as the machine fills up.

$350
Fixed cost / month
2× RTX 3090 · 48 GB VRAM
12/12
Tool-calling test passed
0 malformed · 0 invented tools
~89
Tokens / sec (single stream)
aggregate scales with batching
$7–9
Cost / user at scale
vs $20–100 on cloud

Cost per user per month

The box cost is fixed. Cost per user is simply $350 ÷ brains served. Optimisation lowers the cost by raising the denominator.

The marginal cost of one more brain is near zero until the concurrency wall. Each added brain lowers the cost for all the others.

Local vs cloud — the crossover

Cloud cost is flat per user. Local cost per user is a falling curve. They cross around 7 users.

Below ~10 brains the cloud still wins (no ops). Past ~20, local takes the lead and the gap widens with scale.

Concurrent brains vs context size

The "1–2 users" figure only holds at full 256K context each. Shrink the context per slot and concurrency multiplies.

Almost no real turn needs 256K. At 64K — plenty for a brain agent — the same box holds ~8 live slots.

VRAM budget (48 GB)

Weights, the per-user KV cache (working memory of context), and overhead all draw from the same 48 GB.

Only ~5 GB free — enough for a small embedding model (RAG), not for a second large model. More VRAM buys more users, not a smarter one.

The throughput myth

89 tok/s is not a fixed pie sliced across users. Single-stream is memory-bandwidth bound; with batching the weights load once and serve everyone, so aggregate throughput rises with concurrency.

80 truly-simultaneous streams at usable speed: no. 80 bursty human users sharing a concurrency-aware engine: plausible. The real ceiling is KV-cache VRAM, not tok/s.