EU AI sovereignty: self-hosted LLMs and the AI Act in practice
What it takes to run LLMs where the data may not leave the EU, and what the AI Act changes for the engineering team.
The problem
When a product sends a prompt to a hosted model API, the prompt travels with everything attached to it: the user's question, the documents retrieved to answer it, and often a customer's personal data. If that API runs outside the EU, the product's customers have a data transfer to account for. If a company subject to US law operates it, they have a second question even when the servers are in the EU.
Two legal facts frame the conversation. Under GDPR (DSGVO in German), personal data may go to a third country only when the conditions of Chapter V are met (Article 44). And the US CLOUD Act of 2018 says a provider's duty to preserve or disclose data under US legal process covers data in its "possession, custody, or control, regardless of whether such communication, record, or other information is located within or outside of the United States" (18 U.S.C. § 2713). That is the sovereignty concern: it is about who can be compelled to hand data over, not where the disk sits.
The questions in a data protection review are predictable, and engineering has to answer most of them:
- Where does inference physically happen, and who operates it?
- Which subprocessors see the prompt, the retrieved documents and the output?
- Does the provider retain any of it, or train on it?
- For a given customer, which model saw which data, and when?
- Can you show the log was not edited afterward?
- How long are prompts and outputs kept, and who decided that?
The last three can only be answered from your own records. No provider contract answers them for you.
The quiet leak is the one teams do not think of as inference. An embedding model called over a US API sees every document you index, the same exposure as sending the documents themselves. So do hosted error trackers that capture request bodies.
The levers that actually work
Self-hosted open-weight models
Running open-weight models such as Llama, Qwen or Mistral on hardware you control removes the transfer question for the prompt. A distinction worth making to customers: the origin of the weights matters far less than where inference runs. Weights running on your hardware send data nowhere.
The common split: Ollama on developer machines, because a model runs with one command; vLLM for serving, because continuous batching and paged attention let many concurrent requests share a GPU instead of queuing behind each other; llama.cpp where there is no GPU, including air-gapped installs. All three expose an OpenAI-compatible API, which is what makes the gateway below possible.
The costs are real. On nCloud, the sovereign cloud platform I run through Napradha, a 1.5B model at 4-bit ran at 13.9 tokens per second on CPUs without AVX-512: enough to prove a pipeline, not to serve users. The GPU tier is a single Tesla T4, passed through to one VM, serving a 7B model at 4-bit through vLLM. One T4 is one T4: a single popular workload would saturate it. Read the licenses too: some carry use restrictions that matter once you ship software into a customer's data center. Measure quality on your own tasks, the same way you would decide whether the cheaper model is good enough.
A gateway in front of every model
Put one entry point in front of every model: local, EU-hosted and customer-supplied. The decisions live there, so no caller can skip them:
- Routing by model, cost and sensitivity, from a table the platform owns. A caller asks for a capability; the platform decides where it runs. The mechanics are in routing between Haiku, Sonnet and Opus.
- Jurisdiction as a property of each route, not of each request. Treat every request as sensitive unless the caller explicitly says otherwise, and let sensitive requests reach only in-perimeter models.
- Quotas: per-tenant token budgets and request rate limits, so one tenant's retry loop does not become everyone's latency.
- Redaction: masking personal data before it reaches a model or a log, where the use case allows it.
On nCloud the gateway enforces tenant authorization, a model allowlist, jurisdiction and per-tenant token budgets, with a request-rate limit per client in front; per-tenant rate limits and redaction are not built yet. On an earlier production Claude platform for an EU FinTech start-up, I built the other half: PII masking, per-tenant rate budgets and routing with model fallback, with zero PII incidents in production logs after rollout.
Audit trails
Record every call, refusals and failures included: who asked, for which tenant, which model and provider answered, what went in and out, token counts, latency and outcome. Make the log tamper-evident: when each record carries its predecessor's hash, editing one breaks verification from that point on. Archive it somewhere the writer cannot quietly overwrite, record deletions in the log itself, and make it exportable in a form an auditor can verify without trusting your software.
A log that answers "what did the model see" contains personal data, so GDPR's data minimization and storage limitation apply to the log as well. Retention is a decision to take with the DPO, not a default in code.
I built this first on nCloud, before choosing any model. The record format, archive, retention and what I would change are in an audit trail for every LLM call. Telemetry belongs beside the audit log, not in place of it; see agent observability.
No US-vendor dependencies in the data path
A self-hosted model still leaks if the install pulls images from a foreign registry at runtime, phones home for a license check, or ships telemetry to a hosted backend. nCloud is built from Apache-licensed open source on rented servers in Germany, under one rule: no US-vendor code in the data path.
The test is installing with the internet severed. On nCloud, the AI platform installed onto a fresh Kubernetes cluster with the internet blocked, from one bundle with every artifact hash-verified: 11 checks passed, 0 failed, 1 skipped. The skipped check was the audit gateway, whose secrets store was outside that test's scope. Air gap means no internet, not no network: DNS, time, a registry and the management API still have to be reachable inside the facility. More on the platform side under Kubernetes for AI.
What the AI Act asks of engineering teams
These dates come from the AI Act on EUR-Lex as amended by the Digital Omnibus on AI, Regulation (EU) 2026/1744, published in the Official Journal on 24 July 2026 and in force since 27 July 2026. This is not legal advice, and whether a system is high-risk is a legal classification.
- Since 2 February 2025: the prohibited practices, and Article 4 on AI literacy. The omnibus reworded Article 4: providers and deployers must take measures to support the AI literacy of their staff, and the text now says this does not require guaranteeing any specific level.
- Since 2 August 2025: obligations for providers of general-purpose AI models.
- Since 2 August 2026: Article 50 transparency. Providers must design systems that interact with people to tell them it is an AI system, unless that is obvious, and providers of systems that generate synthetic audio, image, video or text must mark outputs in a machine-readable format. Systems placed on the market before 2 August 2026 have until 2 December 2026 for the marking duty.
- From 2 December 2027: high-risk systems under Annex III, including automatic logging (Article 12) and keeping those logs for at least six months, by providers (Article 19) and by deployers (Article 26(6)).
- From 2 August 2028: high-risk AI in products covered by Annex I.
For most teams shipping an LLM feature, Articles 4 and 50 are the ones already in force. Build toward the logging duties now, because logs cannot be produced retroactively.
How to measure it
Share of requests that stayed in the perimeter. From the gateway's records: calls served by an in-perimeter route divided by all calls, refusals included. Count embeddings and tool calls, not only chat completions, and break it down per tenant. Anything below 100% should trace to a decision someone recorded.
Cost per request against the hosted API. Self-hosted cost is mostly fixed, so utilization decides it:
self-hosted cost per request = (monthly server cost + operations time) / requests served that month
hosted cost per request = input tokens x input price + output tokens x output price
Use the token counts the gateway recorded for both sides. An idle GPU is the most expensive line on the bill, so design for attaching GPU capacity only while there is work for it. The general method is under LLM cost engineering, with a worked production example in cutting Claude inference costs by 38%.
Whether an audit question can be answered from the logs alone. Run a drill: "show every call that touched customer X's data last quarter, which models answered, and prove the records are unaltered." The bar I set on nCloud is that someone who did not build the system answers it from the export within an hour. Watch the evidence pipeline too, and alert on a job that did not run, not only on one that failed: a scheduled job stuck waiting for a volume produces no failed run and no alert.
Common mistakes
- Treating an EU region as the whole answer. Location does not settle who can be compelled to disclose data. Ask who operates the service, under which law.
- Forgetting the side channels. Embedding APIs, error trackers, observability services and runtime downloads all move data or dependencies across the perimeter.
- Logging only successes. Refused, failed and abandoned calls are what an incident review asks about. Refuse calls with no identifiable caller rather than logging them as anonymous.
- Keeping logs that whoever runs the system can edit. A database table proves logs exist, not that they are unaltered. Chain the records and anchor the head somewhere else.
- Making the sovereignty control opt-in. A sensitivity flag that defaults to off fails silently, because a request that left the EU looks the same as one that did not. I made this mistake on nCloud and fixed it while every route was still local.
- Trusting a green job. On nCloud, archive and retention ran green for a while over a test chain, while the chains recording real inference were archived by nothing. Success over the wrong data looks exactly like coverage.