Exemplars let you jump from a metric aggregation to the exact trace that caused it.

Let’s watch this in action. Imagine we have a Prometheus dashboard showing high latency for a particular API endpoint, /api/v1/users/{id}. We see a spike in http_requests_duration_seconds_bucket for this endpoint, specifically in the 500ms-1s bucket. Normally, we’d then have to go hunting through logs or try to find traces matching that time window and endpoint. With exemplars, Prometheus has already done the heavy lifting.

Here’s a simplified Prometheus query and its output that includes exemplar information:

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "http_requests_duration_seconds_bucket",
          "endpoint": "/api/v1/users/{id}",
          "handler": "/api/v1/users/{id}",
          "instance": "localhost:9090",
          "job": "my-app",
          "le": "1"
        },
        "value": [
          1678886400,
          150
        ],
        "exemplar": {
          "labels": {
            "traceID": "a1b2c3d4e5f67890",
            "spanID": "f0e9d8c7b6a54321"
          },
          "value": "0.75",
          "timestamp": 1678886399
        }
      }
    ]
  }
}

Notice the "exemplar" field. It contains traceID and spanID labels, along with the actual value that caused this data point (0.75 seconds). This is the magic. We can take this traceID and immediately query Jaeger (or any compatible tracing backend) for the trace.

This solves the "needle in a haystack" problem of correlating performance issues seen in metrics with the specific request that caused it. Instead of guessing or broad searching, you have a direct pointer.

How it Works Internally

Prometheus collects metrics. When you configure it to use exemplars, it doesn’t just store the aggregated counts for histogram buckets. It also looks for specific spans from your tracing system that are associated with those metric observations.

  1. Instrumentation: Your application code is instrumented for both metrics (e.g., using Prometheus client libraries) and tracing (e.g., using OpenTelemetry or Jaeger clients).
  2. Metric Observation: When an HTTP request takes 0.75 seconds, your metrics library increments the relevant http_requests_duration_seconds_bucket. Crucially, if tracing is enabled and the current request has a trace context (traceID, spanID), the metrics library attaches this trace context to the metric observation. This is often done by adding specific labels (traceID, spanID) to the metric itself at the point of observation.
  3. Prometheus Scrape: Prometheus scrapes your application’s /metrics endpoint. It ingests the metrics, including the http_requests_duration_seconds_bucket and its associated traceID/spanID labels.
  4. Exemplar Storage: Prometheus identifies these trace-context-enriched metric observations as potential exemplars. It stores a single exemplar per metric time series per time interval (or based on configuration). The exemplar stores the trace ID, span ID, the observed value (e.g., 0.75s), and a timestamp.
  5. Querying: When you query Prometheus for metrics and enable exemplar display (e.g., in Grafana), Prometheus returns the exemplar data alongside the metric values. You then use this exemplar data to query your tracing backend.

The primary problem this solves is debugging performance regressions or anomalies. When a metric shows a problem (e.g., increased latency, higher error rate), you can quickly pinpoint the exact request path and the specific trace that exhibited that behavior. This dramatically reduces the time to identify the root cause, as you’re not sifting through general logs or trying to correlate timestamps across different systems.

You typically configure Prometheus to scrape metrics with exemplar support enabled. For Prometheus itself, this might look like:

scrape_configs:
  - job_name: 'my-app'
    static_configs:
      - targets: ['localhost:9090']
    exemplar:
      enabled: true
      # Optional: filter to only collect exemplars for specific metrics
      # metrics:
      #  - http_requests_duration_seconds_bucket
      #  - grpc_server_handling_seconds_bucket

And in your application’s instrumentation, you’d ensure trace context is propagated to metric labels. For OpenTelemetry, this might involve using OpenTelemetrySDK with both metric and trace providers configured, and ensuring your metric instruments are aware of the current trace context. A common pattern is using a SpanExporter that also injects trace context into metric labels.

The key is that the trace context is attached to the metric observation at the source, before aggregation. Prometheus doesn’t magically link a metric to a trace; the trace information is carried along with the metric data from the application.

What’s often missed is that the sampling strategy of your tracing system directly impacts exemplar availability. If your tracer samples aggressively and doesn’t capture the request that actually caused the metric anomaly, no exemplar will be generated or stored, even if your Prometheus is configured for them. This means you need a tracing sampling strategy that’s sensitive enough to capture the kinds of events you’re interested in correlating with metrics, or you must ensure your metrics are being recorded with exemplar support before any potential trace sampling might discard the request.

The next step is often visualizing these exemplar-linked traces directly within your Prometheus dashboarding tool, like Grafana.

Want structured learning?

Take the full Prometheus course →