Leadership wants "our own ChatGPT." The data team orders a GPU server before answering: what latency, which documents, who accesses, how many requests per day. Three months later: €2,400/month idle GPU, model never connected to IT, DPO discovering prompts contain HR data.
Hosting a language model is not buying a card. It is a chain: model choice, quantization, inference server, embedding storage, observability, prompt security. GPU comes after that mapping.
Constraints to fix before hardware
| Question | Why it is priority |
|---|---|
| Target latency | Under 1s interactive → GPU; overnight batch → CPU possible |
| Context size | 32k tokens multiply video memory |
| Sensitive data | GDPR, medical secrecy → region, logs, prompt retention |
| Request volume | 10/day vs 10,000/day changes architecture |
| Fine-tuning or RAG only | RAG = vector DB; fine-tuning = training GPU |
Common self-host stack
Open-weights model — verify license and commercial use. Inference server — vLLM, Text Generation Inference, llama.cpp by scale. Optional RAG — PostgreSQL/pgvector, Qdrant, separate embeddings. Reverse proxy and auth — no exposed API without rate limits. Logs — prompts may contain PII; retention policy mandatory.
CPU-only: Ollama or llama.cpp on RAM-heavy server for small teams. GPU: plan context margin; multi-GPU for 70B+ models.
Self-host vs managed API
| Criterion | Self-host GPU | Cloud API |
|---|---|---|
| Confidentiality | Full control if well operated | Depends on contract and region |
| Initial cost | High GPU rental | Low, usage billing |
| Operations | Drivers, model updates | Minimal |
| Customization | Fine-tuning, deep RAG | Limited to API options |
For many SMBs, European API plus self-host RAG — data at home, inference at vendor — is the reasonable compromise when legal review accepts a subprocessed model under DPA.
Before any GPU order, define what happens during a two-week hardware outage: fallback API, degraded mode, or hard stop — if you cannot answer, you are not ready for production self-hosting.
The peak: GPU is the last budget line
Compare region and egress on the directory before signing a year of GPU rental.
Decide and move forward without blind spots
Document usage, data sensitivity, latency targets and daily request volume before any hardware purchase. Run a proof of concept on CPU or small GPU with a quantized model and real metrics — not slide decks alone. Validate legal frame: prompts, logs and subprocessors with your DPO. Scale GPU and inference only when the proof of concept has real users and measured load.
Frequently asked questions
GPU mandatory?
Interactive and medium models: yes. Batch or small quantized model: CPU possible for POC.
VRAM for 7B?
4–6 GB quantized, ~14 GB full precision; target 24 GB in production with context margin.
Self-host or API?
Sensitive data or volume → self-host or hybrid RAG. Time-to-market → European API.
Hosting in Europe?
EU region, DPA, prompt log policy — not flag alone.
Before picking a card, one question: what happens if we have no GPU for two weeks? If you have no answer, you are not ready to host an LLM — only to fund one.