EU AI sovereignty & self-hosted LLMs
An audit trail for every LLM call: why I built the evidence layer first
When I built the AI layer of nCloud, the sovereign cloud platform I run through Napradha, the first component was not a model server. It was an audit gateway: one service that every model call passes through, and that writes a tamper-evident record of each call, whatever its outcome.
Models are interchangeable: local open-weight models, EU-hosted APIs and a customer's own endpoint all speak the same OpenAI-compatible chat format. What a customer cannot swap is the answer to their auditor: which model saw which data, on whose behalf, and proof that the log was not edited. That is the part they have to prove, so I built it first.
nCloud is assembled from Apache-licensed open source (CloudStack, managed Kubernetes on it, Ozone for object storage), runs on rented servers in Germany, and follows one rule: no US-vendor code in the data path.
Status: the gateway and its archive and retention jobs run on nCloud, the evidence export is built, and all of it has been tested end to end. There are no customers or customer traffic on it yet, and no console UI: the evidence is an API and command-line tools.
What the gateway does on every call
- Identify the caller. An unauthenticated request is refused, not logged as anonymous. Otherwise a caller keeps themselves out of the log by omitting a header.
- Authorize the tenant through Casbin. Refusals are recorded, because "did anyone try" is what an incident review asks.
- Check policy before retrieval. Model allowlist, jurisdiction and the tenant's token budget are checked before any document is fetched. A refused request must not have caused work.
- Retrieve inside the gateway, not in the caller. Then the recorded context is what the model actually saw, filtered by the tenant the gateway authorized.
- Call the model, then write the record, whatever happened. A timeout, a provider error or a client that disconnects mid-stream all produce a record with an outcome that says so.
- Emit telemetry after the record. An OpenTelemetry span using the GenAI semantic conventions carries the record's sequence number, so a trace resolves to the exact audit entry. Losing a span is survivable; losing the record is not. More on that under agent observability.
- Return the sequence number and hash to the caller, so the client can keep its own copy of the chain head.
What gets recorded, and why
The field list is fixed in code, and every content field is covered by the hash.
| Field | Why it is there |
|---|---|
seq, ts |
Order and UTC time. A gap in seq is how a deletion shows up |
actor, tenant |
Who asked, and on whose behalf. The tenant is authorized on every call |
provider, model |
Where inference happened. The provider label names a place, such as local-gpu, not a brand |
prompt, context, tool_calls, output |
What the model saw and said, including retrieved passages and why each was retrieved |
prompt_tokens, completion_tokens, latency_ms |
Cost and performance per call and per tenant |
outcome |
ok, a refusal such as denied_jurisdiction, a provider error, client_disconnected, or ok_estimated_usage when the server sent no token counts |
digest, prev_hash, hash |
The chain |
An example record: the field names are the implementation's, the values are invented, and nested fields are shown decoded:
{
"seq": 1042,
"ts": "2026-09-18T09:14:03Z",
"actor": "claims-triage-service",
"tenant": "tenant-demo",
"provider": "local-gpu",
"model": "Qwen/Qwen2.5-7B-Instruct-AWQ",
"prompt": [
{"role": "system", "content": "Answer only from these sources and cite them by number. [1] policy-2024.pdf p.3 ..."},
{"role": "user", "content": "Is water damage from a burst pipe covered?"}
],
"context": {
"passages": [{"chunk_id": "c-8812", "filename": "policy-2024.pdf", "page": 3, "text": "..."}],
"retrieval": {"mode": "hybrid", "fusion": "reciprocal-rank",
"selected": [{"chunk_id": "c-8812", "found_by": "lexical+dense", "rrf_score": 0.0323}]}
},
"tool_calls": [],
"output": "Yes. Sudden escape of water from a fixed pipe is covered [1].",
"prompt_tokens": 812,
"completion_tokens": 164,
"latency_ms": 5210,
"outcome": "ok",
"digest": "9f2c4e...",
"prev_hash": "41ab07...",
"hash": "c07e93..."
}
The chain works in two steps:
digest(n) = sha256(canonical_json(record n))
hash(n) = sha256(prev_hash || digest(n))
Editing any field of a record changes its digest, which breaks its hash and every hash after it. The first exit test did exactly that: a record rewritten in place looked normal on inspection and failed verification at that entry.
This was not my first version. The first per-tenant export failed verification, and the verifier was right: one tenant's records are a subset of a shared chain, and a gap looks like a deletion. Chaining over the digest fixed it. Other tenants' records are exported as digest and hashes only, which proves continuity and reveals nothing.
What I do not record, and what needs care
The fixed field list is itself a data minimization control: nothing lands in the record by accident. There is no client IP address, no request headers and no provider API key. Adding a field is a deliberate change, because it changes what the hash covers.
The gateway stores prompt, retrieved context and output in full, because an incident review asks what the system saw and said. That makes the audit log a store of personal data under GDPR, bound by data minimization and storage limitation (Article 5(1)(c) and (e)). The audit report says so on its face.
On an earlier production Claude platform for an EU FinTech start-up, I built PII masking alongside conversation-level audit trails, and there were zero PII incidents in production logs after rollout. The nCloud gateway does not mask yet.
Archive and retention
Two scheduled jobs run for each chain:
- Archive, nightly. It refuses to archive a chain that does not verify. It writes the records as JSON Lines to Ozone, plus an anchor: the head's sequence number and hash, chained to the previous anchor. Rewriting history now means rewriting a second chain in a different system.
- Retention, weekly. It deletes records older than six months, and refuses to run if nothing has been archived or the chain does not verify. The deletion is appended to the chain before anything is removed, and a checkpoint lets the verifier tell lawful deletion from tampering.
The honest limit: the anchors live in nCloud's own object store. Third-party anchoring, such as an RFC 3161 timestamp, is not in place yet.
For an audit, an export produces a bundle: the records, a standalone verify.py using only the Python standard library, the retention policy and deletions, the models, providers and jurisdictions in use, the access policy at export time, the anchors, and human review decisions. I verified one on a stock Python in a directory holding nothing but the bundle. It caught an edited record, a forged digest and a deleted record.
Model tiers behind one gateway
| Tier | What serves it | State on nCloud |
|---|---|---|
| Local GPU | vLLM on a Tesla T4 passed through to one VM, 7B model, 4-bit AWQ | Wired through the gateway, internal test traffic only |
| Local CPU | llama.cpp, small quantized model | Measured at 13.9 tokens/s for a 1.5B 4-bit model on Xeons without AVX-512. Proves the pipeline, not throughput |
| EU-hosted API | A model API run by an EU provider | Designed for, not connected yet |
| Customer-supplied | Their GPU or their endpoint, same wire format | Designed for, no customer yet |
APISIX in front routes by path prefix to one audit gateway instance per provider, and no route reaches a provider directly. Two gates apply to every call, and both default to deny. A model missing from the platform's routing table is refused, and every route must declare a jurisdiction. Every request counts as sensitive unless the caller says otherwise, and a sensitive request may only reach a local route. A non-EU API can be configured where a customer permits it, and every record names the provider, so a report shows which calls left the EU. Cost routing belongs here too, as in routing between Haiku, Sonnet and Opus.
Installable with the internet severed
"Air-gap capable" is an easy claim, so I tested it: a fresh Kubernetes cluster, internet blocked, one bundle, every artifact hash-verified, images from a registry inside the facility. On 2026-08-10 the exit test passed 11 checks, failed 0 and skipped 1.
The skip was the audit gateway itself. It needs the platform's secrets store, which was not part of that test cluster, so it counted as neither pass nor failure. The rest of the AI pipeline installed and ran with no internet, and document extraction served a real request.
Air gap means no internet, not no network: a facility still has DNS, time, a registry and a management API, and the test asserts both halves. It also found two faults no review would have caught: containerd ignored the registry mirror configuration until config_path was set, and the bundle missed components the provisioning code fetched from the internet at install time.
For a sovereignty buyer it is the only real evidence that nothing calls home (more under Kubernetes for AI).
How this maps to the AI Act
This is an engineering reading of the AI Act and the Digital Omnibus on AI, not legal advice.
- Article 12(1): high-risk AI systems "shall technically allow for the automatic recording of events (logs) over the lifetime of the system". Article 12(2) says what the logs serve: identifying risk situations, post-market monitoring, and monitoring of operation.
- Articles 19(1) and 26(6): providers and deployers keep the logs under their control for at least six months, unless Union or national law, in particular data protection law, provides otherwise.
- Dates: the omnibus, Regulation (EU) 2026/1744, was published in the Official Journal on 24 July 2026 and entered into force on 27 July 2026. It moved Chapter III Sections 1 to 3, which contain Articles 12, 19 and 26, to 2 December 2027 for Annex III high-risk systems and 2 August 2028 for Annex I. I found no change to the wording of those three articles.
The Act asks for logs, not a hash chain. The chain is my answer to the follow-up question: show that nobody edited them. Whether a system is high-risk is a legal classification. The rest of the timeline is in the EU AI sovereignty hub.
What I would do differently
- Put real inference through it sooner. The first gateway used a stand-in provider that returns a character count, so the chain could be tested before any model was chosen. For a while the archive and retention jobs covered only that test chain and reported success nightly, while the chains recording real inference were archived by nothing. A green job over evidence nobody will ask for looks exactly like coverage.
- Deny by default from the first commit. The sensitivity flag first defaulted to off, so the jurisdiction gate fired only when a caller asked for it. I flipped it while every route was still local, so no live call changed. It should never have been opt-in.
- Decide minimization per field before the first record: masking or pseudonymization at the gateway, and data subjects tagged at ingestion. Today a subject search over the evidence is a best-effort content search, and the export says so.
- Anchor outside my own infrastructure from the start. It turns "hard to rewrite" into "a rewrite someone else can detect".
- Build the evidence view with the evidence. The data behind a console exists; the screens do not, and a compliance officer will not read JSON.
- Run the air-gap test with the gateway in scope, so the component the product is sold on is not the one the test skipped.