Kubernetes for AI workloads

One OpenTelemetry layer across four clouds: moving off ELK without a coverage gap

For an airline group, I replaced ELK-based log collection with OpenTelemetry as the one collection layer of a platform that ran end-to-end operations for five airlines. Three decisions carried it: fix collection before touching the backend, do the data reduction in the OpenTelemetry Collector close to the source, and run the new pipeline beside the old one until it was proven, then cut over service by service.

Where it started

The platform was live and mission-critical, and it ran on four clouds: AWS, GCP, Azure and Oracle Cloud. Each had grown organically with its own CI/CD and its own observability, and configuration drift between them caused recurring outages. The wider fix was one portable architecture on Kubernetes and Helm across all four providers, with one observability stack. This post is about the observability part.

Monitoring was built on the ELK stack, and it was struggling. Log collection was heavy on the network. Ingestion could not keep up at peak, which is when an incident is most likely. And there were no unified traces, so correlating an issue across services meant jumping between logs. On a live airline system, visibility that arrives late is an operational risk, not an inconvenience.

The task was an observability layer that could see all five airlines' workloads in real time, with metrics, logs and traces in one place, less overhead on the network, and a vendor-neutral foundation that could evolve.

Fix the collection layer, not the backend

The tempting move is to swap the backend. It is the wrong first move. The problem was not where the data ended up; it was how much raw data crossed the network to get there.

So OpenTelemetry became the collection standard, and the backends became replaceable parts behind it:

• services instrumented with the OpenTelemetry SDKs and auto-instrumentation, so every service emitted metrics, logs and traces in the same shape
• OpenTelemetry Collectors close to the source, each running the receivers → processors → exporters pipeline
• Grafana for dashboards, Prometheus for metrics, and traces for cross-service correlation

Once OpenTelemetry is the collection layer, changing a backend means changing an exporter in the Collector configuration, not re-instrumenting every service. That is where the reduction in lock-in comes from.

Doing the work in the Collector

The Collector is where the network problem was solved. Three processors did most of it:

• batching, so data leaves in fewer, larger, better-compressed requests
• tail-based sampling for traces: keep every error and every slow trace, and sample the healthy, fast majority
• attribute filtering, dropping high-cardinality, low-value attributes before they cost bandwidth and storage

Cutting low-value data before it was shipped is what relieved the network and the ingestion pressure, and log-collection performance and network efficiency improved a great deal as a result.

Here is the shape of the sampling tier as an illustrative configuration. It is an example, not the production file, and every threshold in it is a placeholder to tune. Component names are as of Collector v0.161.0:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  attributes/drop-noise:
    actions:
      - key: url.query            # high cardinality, low value
        action: delete
      - pattern: ^debug\..*
        action: delete
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 2000 }
      - name: sample-the-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }
  batch: {}

exporters:
  otlp_grpc:                      # named "otlp" in older Collector releases
    endpoint: traces-backend:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, attributes/drop-noise, tail_sampling, batch]
      exporters: [otlp_grpc]

The order matters. The batch processor's own documentation says to put it after memory_limiter and after any sampling, so you do not spend effort batching data you are about to drop.

If you copy this, one constraint decides your topology. Tail sampling decides per trace, so the tail sampling processor needs every span of a trace to reach the same Collector instance, and a trace that crosses services usually crosses nodes. The documented answer is two layers: Collectors near the source that batch, filter and forward through the load-balancing exporter, routed by trace ID, and a second layer that runs tail sampling. Head sampling in the SDK needs no such topology, but it decides before it knows whether the request will fail, and the failed request is exactly the trace you want to keep.

Five airlines, one vocabulary

I set semantic conventions so all five airlines' data was consistent and comparable, with one shared Collector configuration and set of processors, and per-airline attributes to keep each airline's data separable.

Without that, "compare airline A with airline B" starts with a translation table. Semantic conventions sound like paperwork until the first cross-tenant question, and then they are what makes the answer one query instead of five.

A tenant attribute is also the seam for everything per-tenant that comes later: filters, per-tenant dashboards, per-tenant alert routing. It belongs on the resource, set in one place, not typed by hand in each service.

Cutting over a system that cannot stop

The new pipeline ran alongside ELK first. The hardest part of the cutover was validating dashboard and alert parity before switching. Only when they agreed did a service move, one at a time, so there was never a moment without coverage on a live system.

Keeping both pipelines live, even briefly, costs money. A monitoring gap on an airline operations system costs more. The run is shorter if parity is defined per service before it starts: which dashboards and which alerts must match, and over what window.

What it gave the platform

• real-time, unified metrics, logs and traces across all five airlines
• distributed tracing that made real-time cross-service debugging possible, and much faster root-cause analysis than jumping between logs
• one observability stack across all four providers
• a vendor-neutral foundation: the backends became swappable, which reduced lock-in

What carries over to inference workloads

That platform was not an AI system, but the pattern transfers almost unchanged to model inference on Kubernetes, and it is how I would instrument one today.

An LLM call is a span. OpenTelemetry's GenAI semantic conventions name the attributes: gen_ai.request.model, gen_ai.provider.name, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. As of 2026-09-19 they live in their own repository and are still marked Development, so pin the version you emit and expect renames.

With the token counts on the span and a tenant attribute on the resource, cost per tenant becomes a query rather than a spreadsheet. The same tail-sampling policy fits too: keep every failed and every slow model call, and sample the rest.

Prompt and completion content is opt-in in those conventions, and it should stay off spans by default. It is the highest-cardinality and most sensitive attribute you have.

Sampling is right for telemetry and wrong for evidence. If you must be able to show what a model was asked and what it answered, that is an audit trail, and it cannot be sampled: see an audit trail for every LLM call. The wider picture of running inference for many tenants is in the Kubernetes for AI workloads hub, and tracing agents and model calls in agent observability.

What I would do differently

Put a cardinality budget into the conventions from day one. The conventions said what each attribute meant. I would also have them say which attributes each signal may carry and which of those may become metric labels, and enforce that in the shared Collector configuration. Filtering high-cardinality attributes at the edge works, but it is a cleanup. A budget stops them being emitted in the first place, and it gives each team a rule to check against instead of a Collector that silently drops their field.

Record the baseline in numbers before the first Collector ships. Telemetry bytes per node per signal, ingestion lag at peak, and time from an alert to the first useful trace, measured the same way before and after. That turns the result into a number anyone can check later rather than an adjective in a retrospective.

Test the Collector configuration like code. One shared configuration across five airlines and four clouds means one bad processor change can drop data for all of them at once. I would run the Collector's validate command in CI, and replay a recorded sample of telemetry through each new configuration to check that errors and slow traces still come out the other end.