Kubernetes for AI workloads: multi-tenant inference in production
Running model inference on Kubernetes for many tenants, without one tenant's traffic or bill landing on another.
The problem
Inference looks like a web service: a request comes in, a response goes out. Kubernetes will run it like one unless you tell it otherwise. Three things are different.
The scarce resource is the GPU, and it comes in whole units. By default a vendor device plugin exposes each GPU as a countable resource, a pod asks for nvidia.com/gpu: 1 in its limits, and it gets the whole device. An idle GPU is the most expensive idle thing in the cluster. A missing one is a queue.
Models are slow to start. A new replica needs a node with a GPU, a container image that is often several gigabytes, and model weights loaded into GPU memory before it can answer anything. That can take minutes where a web pod takes seconds, and every scaling and rollout decision has to account for it.
Tenants are noisy. One tenant's burst can fill the shared queue or saturate the shared GPU, and without per-tenant accounting its cost lands on everyone's bill, or on no one's.
I have run multi-tenant AI workloads on two platforms on AWS EKS. For an audio-commerce start-up, customers of local Indian shops order by speaking Tamil or English, and Whisper microservices on EKS transcribe those orders for many shops on one platform. For an EU FinTech start-up running a second-hand marketplace, the model sat behind the Claude API, so what ran on Kubernetes was the platform around it: an async worker pool and the isolation between buyer, seller and moderator workspaces, not GPUs. Both needed a predictable cost per unit of work. On nCloud, my own platform, a Tesla T4 is passed through to one VM for model serving, which keeps GPU scarcity concrete.
The levers that actually work
Tenant isolation
On both EKS platforms, tenant isolation was OIDC for identity, namespace RBAC for access, and per-tenant rate limits or budgets. On the voice platform, each shop mapped to a workspace.
That is soft multi-tenancy. The Kubernetes multi-tenancy documentation uses "hard" multi-tenancy for tenants that do not trust each other. Namespaces scope names, RBAC roles and network policies. Tenants still share nodes, the kernel, the control plane and, if you let them, the GPU.
Namespaces were enough there because a tenant was a workspace or a shop inside one product: every container on the cluster ran code the platform shipped. The risk was not a hostile neighbor but a loud one, and the per-tenant rate budget is what dealt with that.
They stop being enough when a tenant brings its own code or model weights, when a contract or a regulator requires a separation you can demonstrate, or when GPUs are shared by time-slicing. Then the boundary moves to dedicated node pools per tenant, and for hard isolation to a separate cluster on the tenant's own VMs. A VM boundary is enforced by the hypervisor. A namespace boundary is policy on a shared kernel.
GPU scheduling, sharing and CPU fallback
Put GPU nodes in their own node pool, taint them, and give only inference pods the toleration and node affinity to land there. Otherwise a log shipper or batch job lands on the most expensive node you have.
Sharing one NVIDIA GPU between pods has two mainstream options, and they are not equivalent. Time-slicing lets several pods take turns on one device, and NVIDIA's documentation is explicit that there is no memory or fault isolation between them. MIG partitions the GPU into instances with memory and fault isolation in hardware, but only on Ampere-generation and later GPUs such as the A100, A30 and H100. A T4 is older, so it cannot do MIG: time-slicing it is acceptable for one owner's models and not for two tenants'.
Dynamic Resource Allocation is the newer route: a pod claims a device by its attributes instead of requesting a counted resource. The Kubernetes documentation marks it stable since v1.35 (checked 2026-09-19). What it does for your GPUs depends on the vendor's DRA driver.
CPU fallback is easy to overlook. The voice platform's Whisper pipeline had GPU/CPU fallback to trade cost against latency, and together with routing and budgeting it is what made cost per audio minute predictable. The general practice: size GPU capacity for the latency-sensitive path, and send work that can absorb more latency to cheaper CPU nodes instead of queuing it for a GPU or forcing a new GPU node. Which requests can absorb it is a product decision: write it down as a latency budget per request type.
Autoscaling on the right signal
CPU is the wrong signal for GPU inference. A GPU-bound pod can sit at low CPU while its queue grows, so a CPU-based HorizontalPodAutoscaler never scales. Scale on signals that track the real bottleneck: queue depth or the age of the oldest message, requests in flight per replica, or GPU utilization from NVIDIA's DCGM exporter through Prometheus. KEDA has scalers for AWS SQS, Redis Streams and Prometheus (docs for version 2.20), which covers most of these.
Both of my EKS platforms put a queue in front of the workers: Redis Streams feeding a Kubernetes worker pool for async chat processing on the marketplace, and S3 → Lambda → SQS → EKS on the voice platform. A queue is the most honest scaling signal: it measures work waiting, not work being done.
Two layers have to agree. The pod autoscaler adds replicas, and a node autoscaler adds GPU nodes when those replicas cannot be scheduled; on the voice platform that was Karpenter. Scale-up time is node provisioning plus image pull plus model load, so trigger early on backlog, and keep a warm floor of replicas for tenants with latency commitments rather than scaling to zero.
Rolling out a model version safely
A new model version is a deployment with two extra ways to fail: it loads slowly, and it can be up but worse.
For the first, readiness has to mean the weights are loaded and a warm-up inference has succeeded, not that the process is listening. Give slow loaders a startup probe with a long enough window, so the liveness probe does not kill a pod that is still loading. Check GPU headroom before a rolling update: maxSurge: 1 needs a free GPU for the surge pod, and without one the rollout stalls or you give up capacity with maxUnavailable.
For the second, evaluate the new model before it takes traffic, as described in is the cheaper model good enough. Then send it a slice of traffic with the model version as an attribute on every span and metric, so old and new can be compared per tenant.
On the marketplace platform, prompts, agents and routing logic went through GitOps CI/CD with gated releases, deployed with ArgoCD. On the voice platform, zero-downtime deploys made daily prompt and pipeline iteration possible. Treat a model version, a prompt and a routing rule as the same kind of change: versioned in git, gated, and reversible.
OpenTelemetry for LLM workloads
One tracing layer across every service is what turns "the model is slow" into a specific span. For an airline group I moved a platform running five airlines on four clouds to one OpenTelemetry layer, with sampling and filtering in the Collector; the details are in one OpenTelemetry layer across four clouds. For a US federal research agency's public challenge, I simulated 280 live camera streams and 60 audio streams as data sources, with end-to-end tracing so every stream was traceable.
For model calls, OpenTelemetry's GenAI semantic conventions define attributes for the model and its input and output tokens. They are still marked Development (checked 2026-09-19), so pin the version you emit. Add the tenant on the resource and cost per tenant becomes a query. Keep prompt content off spans: it is opt-in in the conventions for good reason. Sampling suits telemetry, not evidence; an audit trail for every LLM call covers the part you cannot sample. The wider observability picture is in agent observability.
How to measure it
| Metric | Why it matters | Where it comes from |
|---|---|---|
| p95 latency per tenant and per model version | An average hides the tenant being starved | Spans or request histograms carrying tenant and model attributes |
| Queue depth and age of the oldest item | The earliest sign that scaling is behind | Queue metrics (SQS, Redis Streams) |
| GPU utilization and GPU memory used | Low utilization with a growing queue means batching or placement is wrong; memory tells you whether a second model fits | DCGM exporter |
| Cost per unit, per tenant | Per request, per 1,000 tokens or per audio minute: the number a tenant's price has to cover | Token counts or audio duration, priced per unit |
| Cold-start time | Sets how early you must scale | From scale event to first successful inference, split into node, image and model load |
| Quality next to cost | A cheaper path that is worse is not cheaper | Evaluation results joined on model version |
On the voice platform, the dashboards linked audio duration to cost to accuracy, and that chain is the one to copy. Cost per audio minute on its own invites a cheaper, worse pipeline.
On the marketplace platform, per-tenant token-spend dashboards made cost visible per workspace. The work there cut Claude inference cost by 38% with quality held; the story is in cutting Claude inference costs by 38%, the lever that gave the most is in routing between Haiku, Sonnet and Opus, and the wider method is in LLM cost engineering.
Common mistakes
- Autoscaling GPU inference on CPU. The pods look idle while the queue grows.
- A readiness probe that passes before the model is loaded. The Service routes traffic to a pod that can only fail or time out.
- Calling namespace isolation hard multi-tenancy. It is soft. That is fine for workspaces inside one product and not for tenants running their own code, so be clear about which one you are offering.
- Averages instead of per-tenant p95. One tenant's bad hour disappears into the mean.
- No per-tenant rate budget. The noisiest tenant then sets everyone's latency and everyone's bill.
- Scaling to zero a model that takes minutes to load. The first request after a quiet period pays the whole cold start.