Metrics tell you that something is slow. Logs tell you what happened on one service. Neither tells you where the time actually went across a request that hopped through six microservices before it failed. That’s the gap distributed tracing fills, and Jaeger is the CNCF project I reach for whenever I’m debugging latency in a Kubernetes-based microservices architecture rather than a monolith.
Why Tracing Is Different From Metrics and Logs
A trace represents one request’s full journey through your system, broken into spans — each span is a unit of work (an HTTP call, a database query, a cache lookup) with a start time, duration, and parent-child relationship to other spans. Jaeger collects these spans, correlates them by a shared trace ID, and lets you visualize the entire request path as a timeline (a “flame graph” / waterfall view).
Request
└─ Span: API Gateway (12ms)
└─ Span: Auth Service (3ms)
└─ Span: Order Service (85ms)
└─ Span: Database Query (78ms) ← the actual bottleneck
└─ Span: Inventory Service (4ms)
Without tracing, “Order Service is slow” is where the investigation stops. With tracing, you immediately see the 78ms database query as the real culprit.
Jaeger Architecture on Kubernetes
Jaeger has several deployable components:
- Agent (optional in newer versions) — receives spans from application SDKs over UDP, batches and forwards them
- Collector — receives spans (via agent or OTLP directly), validates, and writes to storage
- Query — serves the Jaeger UI and API, reading from storage
- Storage backend — Elasticsearch, Cassandra, or (for small setups) in-memory
App (OpenTelemetry SDK)
│ OTLP/gRPC
▼
Jaeger Collector
│
▼
Storage (Elasticsearch/Cassandra)
│
▼
Jaeger Query ──▶ Jaeger UI
Installing the Jaeger Operator
The Jaeger Operator manages the lifecycle of Jaeger instances via a CRD, which is far less error-prone than hand-deploying each component.
kubectl create namespace observability
kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.57.0/jaeger-operator.yaml -n observability
Verify:
kubectl get pods -n observability
Deploying a Jaeger Instance
For production, back Jaeger with Elasticsearch rather than the default in-memory storage (which loses all traces on pod restart):
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: production-jaeger
namespace: observability
spec:
strategy: production
storage:
type: elasticsearch
options:
es:
server-urls: http://logging-es-es-http.logging.svc:9200
index-prefix: jaeger
ingress:
enabled: true
collector:
replicas: 2
resources:
limits:
cpu: "1"
memory: 1Gi
query:
replicas: 1
kubectl apply -f jaeger.yaml
kubectl get jaeger -n observability
kubectl get pods -n observability
For local testing or small clusters, strategy: allInOne is a single-pod deployment good enough for evaluation but not production:
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: simple-jaeger
namespace: observability
spec:
strategy: allInOne
Instrumenting Applications with OpenTelemetry
Jaeger’s modern ingestion path is via OpenTelemetry (OTLP), not the legacy Jaeger-native client libraries, which are deprecated. Here’s a minimal Node.js instrumentation example:
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const sdk = new NodeSDK({
traceExporter: new OTLPTraceExporter({
url: 'http://production-jaeger-collector.observability.svc:4317',
}),
instrumentations: [getNodeAutoInstrumentations()],
serviceName: 'order-service',
});
sdk.start();
For applications you can’t easily instrument directly, the OpenTelemetry Operator can auto-inject instrumentation via a sidecar/init-container pattern:
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: auto-instrumentation
namespace: production
spec:
exporter:
endpoint: http://production-jaeger-collector.observability.svc:4317
propagators:
- tracecontext
- baggage
sampler:
type: parentbased_traceidratio
argument: "0.1"
Then annotate your Deployment’s pod template to opt in:
metadata:
annotations:
instrumentation.opentelemetry.io/inject-nodejs: "true"
Sampling Strategy
Tracing every single request at scale is expensive in storage and processing. A 10% sampling rate (as set above with traceidratio: 0.1) is a common production starting point, but I always keep error traces at 100% via tail-based sampling if using the OpenTelemetry Collector as an intermediary:
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: otel-collector
namespace: observability
spec:
config: |
receivers:
otlp:
protocols:
grpc: {}
processors:
tail_sampling:
policies:
- name: errors-policy
type: status_code
status_code: {status_codes: [ERROR]}
- name: probabilistic-policy
type: probabilistic
probabilistic: {sampling_percentage: 10}
exporters:
otlp:
endpoint: production-jaeger-collector.observability.svc:4317
service:
pipelines:
traces:
receivers: [otlp]
processors: [tail_sampling]
exporters: [otlp]
Accessing the Jaeger UI
kubectl port-forward -n observability svc/production-jaeger-query 16686:16686
Open http://localhost:16686, select a service, and search traces filtered by duration, tags, or error status. The flame graph view is where you’ll spend most of your debugging time — it makes N+1 query problems and unnecessary serial calls (that should be parallel) immediately visible in a way logs never will.
Correlating Traces with Metrics and Logs
The real power shows up when trace IDs are injected into your structured logs and exposed as exemplars in Prometheus. Grafana supports jumping from a Prometheus latency graph directly into the corresponding Jaeger trace via exemplar links — set this up by ensuring your OpenTelemetry SDK adds trace_id to log output and configuring Grafana’s Jaeger datasource with trace-to-logs correlation.
Securing Jaeger in Production
Trace data often contains request parameters, headers, and sometimes accidentally-logged sensitive fields — treat it with the same care as application logs:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: jaeger-query-ingress
namespace: observability
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/auth-secret: jaeger-basic-auth
spec:
tls:
- hosts:
- jaeger.internal.example.com
secretName: jaeger-tls
rules:
- host: jaeger.internal.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: production-jaeger-query
port:
number: 16686
For genuinely sensitive fields, add a span processor in your OpenTelemetry Collector pipeline that strips or redacts specific attribute keys before export:
processors:
attributes:
actions:
- key: http.request.header.authorization
action: delete
- key: user.email
action: delete
Troubleshooting Missing or Broken Traces
Traces not appearing in Jaeger UI at all. Check the collector is actually receiving spans:
kubectl logs -n observability -l app=jaeger -l app.kubernetes.io/component=collector --tail=50
Verify your application’s OTLP exporter endpoint matches the collector’s actual Service DNS name and port — a surprisingly common mistake is pointing at port 4317 (gRPC) when the app is configured for HTTP export on 4318, or vice versa.
Traces appear but are missing spans from downstream services. This is almost always a broken context propagation — check that every service in the chain is using the same propagator format (tracecontext is the W3C standard and what most modern instrumentation defaults to) and that any message queue or async boundary explicitly carries the trace context in message headers/metadata, since it doesn’t propagate automatically across queues.
High cardinality tags blowing up storage. Avoid putting unbounded values (raw user input, full request bodies, timestamps) directly as span tags — use them as span logs/events instead if you need them for a specific debugging session, and strip them from the default instrumentation.
Collector OOMKilled during traffic spikes. This is why tail-based sampling processors need a decision_wait buffer with bounded memory:
processors:
tail_sampling:
decision_wait: 10s
num_traces: 50000
expected_new_traces_per_sec: 1000
Tune num_traces down if you’re consistently hitting memory limits — this trades a slightly higher chance of dropped decisions for collector stability, which is almost always the right tradeoff during an actual incident.
Production Best Practices
- Run the collector with multiple replicas behind a Kubernetes Service for HA — a single collector pod dying shouldn’t drop all in-flight traces.
- Set explicit resource requests/limits on collector pods; trace volume spikes during incidents (exactly when you need tracing most) can otherwise OOM the collector.
- Use index lifecycle management on the Elasticsearch backend, same as with logging — trace data grows fast and old traces are rarely useful after a couple of weeks.
- Secure the Jaeger UI behind authentication (via Ingress + OAuth2 proxy or similar) — trace data can contain sensitive request parameters.
Real-World Debugging Example
Here’s a concrete case from a service I once debugged: an order-confirmation endpoint had p99 latency around 900ms with no obvious cause in metrics dashboards — CPU, memory, and error rate all looked fine. Pulling up a slow trace in Jaeger’s flame graph immediately showed the actual shape of the problem: three sequential, avoidable calls to the same inventory service for different SKUs in the same order, each taking roughly 250ms, when they could have been batched into one call or run in parallel.
kubectl port-forward -n observability svc/production-jaeger-query 16686:16686
Searching for traces with duration > 800ms and the relevant service name surfaced this pattern within minutes — something that would have taken hours of log correlation across services to piece together manually. This is really the core value proposition of tracing: it doesn’t just tell you something is slow, it shows you the actual causal structure of why.
Common Mistakes
- Using the deprecated Jaeger client SDKs instead of OpenTelemetry — new projects should standardize on OTel from day one.
- Sampling at 100% in production without tail-based sampling, leading to massive storage costs for traces nobody looks at.
- Forgetting to propagate trace context across async boundaries (message queues, background jobs), which breaks the trace chain and leaves you with disconnected fragments instead of one coherent trace.
- Treating tracing as a replacement for metrics and logs rather than a complement — each of the three observability pillars answers a different question, and skipping the other two makes tracing far less useful in isolation.
Summary
Jaeger closes the observability gap that metrics and logs can’t — seeing exactly where time is spent across a distributed request path. On Kubernetes, the Jaeger Operator plus OpenTelemetry instrumentation (ideally via the OpenTelemetry Operator’s auto-injection) gets you there with minimal hand-wiring. The two decisions that matter most in production are your storage backend (Elasticsearch, properly lifecycle-managed) and your sampling strategy (low base rate, 100% on errors via tail-based sampling).
