Observability with OpenTelemetry: one instrumentation for AWS, Azure and your own backends
Instrument services once, send the data wherever the environment needs it, keep the telemetry bill in proportion, and change backends without going blind.
The problem
Most systems become observable by accident. Each service got the logging library its team liked, metrics came from whatever the platform exposed, and tracing arrived later, added by one team with whichever vendor SDK was current that year. It holds up until an incident crosses a service boundary. Then the trace stops at the first queue, two services log the same request under different field names, and the dashboard everyone trusts turns out to count something slightly different from what its title says.
Three costs follow from that.
- Time. Correlating an incident by hand, across logs, metrics and a partial trace, is slowest exactly when speed matters most.
- Money. Telemetry volume grows with traffic, not with team size, so the bill rises fastest when the business is doing well. Much of what gets shipped and stored is never queried.
- Lock-in. Once application code calls a vendor's SDK directly, changing backend means changing code in every service. That is why so many teams stay on a platform they no longer like.
OpenTelemetry deals with all three by separating the parts. Services are instrumented once, against a vendor-neutral API. A Collector receives the data, processes it and exports it. The backend becomes a decision you can revisit: AWS X-Ray and CloudWatch, Azure Monitor, Prometheus and Grafana, Jaeger, or more than one at a time.
I have built observability this way on an airline group's operations platform spread across four clouds, on a streaming simulation for a US federal research agency's public challenge, on a Berlin software company's move from a monolith to microservices, and on an audio-commerce platform on Amazon EKS. The patterns below come from that work.
The levers that actually work
Agree the vocabulary before the first span
The most valuable hour in an observability project is spent before anyone writes code: write down the semantic conventions. OpenTelemetry publishes standard attribute names for HTTP, databases, messaging and more, and those should be used as published. What it cannot define is your domain: the tenant, the customer tier, the business transaction. Decide those names once, decide which signal may carry which attribute, and keep the list in the repository next to the Collector configuration.
On the airline platform, five airlines ran on shared services. A shared vocabulary with a per-airline attribute made their data comparable and let each airline's view be filtered cleanly. Without it, five teams would have invented five spellings of the same field, and every dashboard would have needed five queries.
Instrument with the OpenTelemetry SDK, not a vendor's
Application code should call the OpenTelemetry API and nothing else. Auto-instrumentation covers frameworks and clients, which in Python means the packages for FastAPI, requests, botocore, SQLAlchemy and similar. Manual spans cover the business steps no library knows about. Custom metrics go through the same SDK. Where the data is sent is configuration, not code.
The part that breaks in practice is context propagation. Inside one process, the SDK carries the trace for you. Across a network hop the context travels in headers, and OpenTelemetry's default is W3C Trace Context (the traceparent header) plus Baggage. Two kinds of hop lose it routinely:
- Queues and events. A message on SQS or Kafka, or an event that triggers a Lambda function, carries no HTTP headers. Put the context into the message attributes when sending, and start the consumer's span from it. On the audio-commerce platform, processing ran S3 → Lambda → SQS → EKS, and every one of those arrows is a place where a trace can quietly end.
- Managed AWS components. Load balancers and Lambda add their own
X-Amzn-Trace-Idheader. For services to join traces that AWS started, configure the X-Ray propagator next to W3C Trace Context. In Python that isOTEL_PROPAGATORS=tracecontext,baggage,xray, with the AWS X-Ray propagator package installed. Without it, the trace splits at the first managed component.
Do the heavy work in the Collector
The OpenTelemetry Collector is a pipeline of receivers, processors and exporters, described in YAML. That makes it the right place for everything that should not be the application's concern:
- Batching, so exports are few and large rather than many and small.
- Tail-based sampling, which decides after a trace has finished. Keep every trace with an error and every trace over a latency threshold, and keep a small share of the rest. Head sampling decides before anything has happened, so it drops exactly the traces you will want later.
- Attribute filtering, which drops fields nobody queries and masks personal data before it leaves the node.
- Routing, which sends traces to one backend and metrics to another, or the same data to two backends during a migration.
On the airline platform, moving this work into Collectors close to the source is what relieved the network and ingestion pressure the old log shipping had caused. The detail, including the Collector pipeline and the cutover, is in one OpenTelemetry layer across four clouds.
One instrumentation, several backends
Because export is configuration, the same instrumented service can report to whichever backend its environment prefers.
- AWS. The AWS Distro for OpenTelemetry (ADOT) packages the Collector with exporters for X-Ray traces and for CloudWatch metrics, which arrive as structured log events in Embedded Metric Format.
- Azure. The Azure Monitor OpenTelemetry exporter sends the same signals to Application Insights.
- Your own infrastructure. Prometheus, Grafana, Jaeger or Tempo, and Loki cover the same ground, and nothing leaves your network.
On the audio-commerce platform, Python services on EKS sent traces to X-Ray and metrics and structured logs to CloudWatch. The team kept working in the AWS consoles it already knew, and the instrumentation stayed neutral. On the airline platform, one layer served AWS, GCP, Azure and Oracle Cloud at once. The point is not that you will change backend often. It is that you can, and that changes every conversation with a vendor.
Migrate with a parallel run, not a cutover date
Replacing the observability stack of a live system is risky for a specific reason: the old stack is the thing that would tell you the new one is failing. So do not switch on a date. Run the new pipeline beside the old one, compare dashboards and alerts over the same period until they agree, then cut over one service at a time, keeping the old path until each service is proven.
That is how the airline platform moved from ELK to OpenTelemetry without a gap in coverage. The parity check is the step teams skip, and it is the only evidence that the new system sees what the old one saw.
How to measure it
Five numbers show whether observability works and whether it is worth what it costs:
- Trace completeness. The share of traces that reach every service they should, rather than stopping at a queue or a managed hop. Broken traces are the most common silent failure, and nothing alerts on them unless you measure them.
- Instrumentation coverage. The share of services emitting all three signals with the agreed conventions. A dashboard over most of your services quietly misrepresents the rest.
- Time to diagnose. From the alert to the service and the change that caused it. Measure it on real incidents and in drills, before and after each change to the pipeline.
- Cost per unit of useful telemetry. Ingest and retention cost divided by something stable, such as requests served or active tenants. An example with made-up numbers: a platform ingesting 2 TB a month at $0.50 per GB pays about $1,000 a month for ingest alone. If tail sampling and filtering remove half of it without losing an error trace, the saving is real and the evidence that nothing important was lost is written down.
- Alert parity during a migration. For each alert in the old system, whether the new one fired for the same events over the same period. Every alert should match before the old path is removed.
Keep the health of the pipeline itself beside those: dropped spans, exporter queue length, Collector memory. A telemetry pipeline that drops data under load fails silently, and at the worst possible moment.
Common mistakes
- Starting with the backend. Choosing the vendor first and instrumenting to fit it is how lock-in gets built.
- Calling a vendor SDK from application code. Every later change of backend becomes a code change in every service.
- Head sampling everything at a flat rate. It throws away most errors along with most successes.
- High-cardinality metric labels. User IDs, request IDs and URLs with IDs in them belong on spans and logs. As metric labels they multiply the number of series, and the bill with them.
- Logs without trace IDs. A log line that cannot be joined to its trace hands the correlation back to a person, during an incident.
- No conventions before rollout. Renaming attributes after dashboards and alerts depend on them costs far more than agreeing them first.
- Cutting over without proving parity. The old system's last useful job is to show that the new one sees the same things.
AI workloads need the same foundations plus a few more signals: token usage, which model answered, and whether the answer was right. That is the subject of agent observability.