Friday evening, a customer clicks "Pay". The request crosses nginx, the checkout API, the inventory service over gRPC, Redis, PostgreSQL, then an async webhook job. Support receives "timeout". API logs show upstream timed out; inventory logs look normal by volume. Yet one slow gRPC call — lost in the average — explains the wait. Without a unified trace, nobody sees it.
Once an action crosses two or more components, diagnosis fragments. Distributed tracing attaches a trace_id to the originating request and records each segment (span): duration, service, error. You see the full tree, not isolated leaves. The real question: can you rebuild the thread a user experiences as a single action?
Traces, logs and metrics: three pillars, not three substitutes
Prometheus will flag a doubled p95 on /api/checkout — see Prometheus: choosing metrics that explain slowness. Centralized logs show a stack trace at a given instant. Neither links the nginx timeout to the slow catalog SQL query, nor to the context-less webhook job.
| Pillar | What it reveals | Common limit |
|---|---|---|
| Metrics | Trends, alerts, SLO compliance | Aggregates — a rare case vanishes in p95 |
| Logs | Textual context, parameters, errors | One silo per service, manual correlation |
| Traces | Full path of one request | Storage cost if sampling is wrong |
Removing traces means guessing where the journey broke — the blind spot that feeds alert fatigue.
Spans, traces and context propagation
A trace represents the full journey of a user action — for example checkout-abc123. A span is a unit operation within that journey: an HTTP GET /stock, a SQL SELECT, a Redis publish. Each span records its duration, status and the service that executed it.
Context propagation is the most frequent weak link. The trace_id must travel from the reverse proxy to async workers via W3C HTTP headers (traceparent, tracestate), gRPC metadata, or the job payload in a queue. Without transmission, the worker creates an orphan trace with no link to the original request — and half the story disappears at the first deferred processing step.
| Component | Common instrumentation |
|---|---|
| nginx | OpenTelemetry module or service mesh ingress |
| PHP / Laravel | open-telemetry/opentelemetry-php library |
| Go / Java | mature native SDKs, straightforward integration |
| Queue worker | context extraction from the job message |
Instrumenting the reverse proxy without instrumenting the application shows entry and exit — not what happens in between.
OpenTelemetry, Jaeger and Tempo: choosing your stack
OpenTelemetry (OTel) collects traces, metrics and logs with a single instrumentation layer. Export to Jaeger (standalone UI), Grafana Tempo (object storage, Loki/Prometheus correlation) or a managed service. On a modest VPS, an OTel Collector agent to Tempo Cloud or Grafana Cloud avoids self-hosting Jaeger and Elasticsearch.
Sampling: do not trace all traffic
Tracing every request in production generates data volumes that are hard to store and query. Two strategies dominate.
Head-based sampling decides at entry (e.g. one request in a hundred) — simple, but a rare case may slip through. Tail-based sampling always keeps slow or errored journeys; Grafana Tempo excels here. Configure rules before a traffic spike, not during the incident.
Correlating logs and traces: closing the loop
Inject trace_id into JSON logs ("trace_id": "abc123..."). In Grafana, one click from a slow span opens the log with the stack trace. Incident path: p95 alert → slowest trace → faulty SQL span → exact query in the log. Exclude sensitive data (email, card numbers, patient IDs) from span attributes.
Limits and common pitfalls
Over-granular spans → overhead and unreadable UI. Desynchronised clocks → inconsistent tree (NTP required). Istio traces HTTP between pods, not your SQL without app instrumentation. "Jaeger installed" ≠ "slow checkout diagnosable": start with revenue-critical journeys.
The peak: three dashboards, zero story
Here is what turnkey "observable" stacks often gloss over.
Decide and move forward without blind spots
In one to two days, make a multi-service architecture diagnosable. First identify one critical journey — checkout, login, partner API — and instrument it before the rest. Deploy OpenTelemetry on the reverse proxy, application and workers, then verify context crosses async queues. Configure tail-based sampling to keep error traces and those above your latency threshold. Inject trace_id into JSON logs and test span → log navigation. Simulate a deliberately slow request: if it does not appear in a single end-to-end trace, fix propagation before the next incident.
Compare hosts that support your own agents via the directory and compare tool. If Prometheus shows degradation without locating the faulty layer, read Prometheus: choosing metrics that explain slowness before adding another dashboard.
Frequently asked questions
Traces, logs or metrics — which do I need?
All three complement each other; none replaces the others. Metrics aggregate over time and drive alerts; logs capture a precise moment with textual context; traces link the segments of a single user request across multiple services. Prometheus alone will not tell you which SQL query blocked checkout — a trace will show it at a glance.
Is OpenTelemetry the standard to adopt?
Yes, it is the most durable choice today. OpenTelemetry provides vendor-neutral instrumentation with exporters to Jaeger, Grafana Tempo, Datadog and other backends. Avoid tying your application to a single proprietary SDK — you will likely change visualization tools before you rewrite your code.
How do I propagate context between services?
Use W3C traceparent and tracestate headers from the entry point through to async workers. Every hop — reverse proxy, gRPC call, queue message — must carry the same trace identifier. Without propagation, each service creates an isolated orphan trace and you lose half the story at the first deferred job.
What sampling strategy works in production?
Head-based sampling at 1–10% is often enough for routine traffic. Tail-based sampling — offered by Grafana Tempo — systematically keeps slow or errored traces, balancing storage cost and diagnostic power. Configure these rules before a traffic spike, not during the incident.
The next time an incident crosses three services with no obvious culprit, ask one question: can we follow a single request end to end? If the answer is no, the missing trace_id will cost more than its instrumentation.