OpenTelemetry traces reveal unexpected bottlenecks by showing how requests flow through your services, not just within them.

Let’s instrument a simple Rust web service to see this in action. We’ll use tracing for instrumentation and opentelemetry-stdout to export our traces to the console for now.

use opentelemetry::{
    global,
    trace::{TraceContext},
    sdk::propagation::TraceContextPropagator,
    trace::TraceError,
};
use opentelemetry_sdk::{
    trace::{Tracer, TracerProvider},
    export::trace::SpanExporter,
    runtime::Tokio,
};
use opentelemetry_stdout::ConsoleExporter;
use tracing::{subscriber::Subscriber, util::SubscriberExt};
use tracing_subscriber::{layer::SubscriberExt, Registry};

fn init_tracer() -> Result<Tracer, TraceError> {
    let exporter = ConsoleExporter::default(); // Export to console

    let tracer_provider = opentelemetry_sdk::trace::TracerProvider::builder()
        .with_simple_exporter(exporter)
        .with_batch_exporter(exporter, opentelemetry_sdk::runtime::Tokio) // Use Tokio runtime for batching
        .build();

    let tracer = tracer_provider.get_tracer("my-rust-app");

    global::set_tracer_provider(tracer_provider);
    global::set_text_map_propagator(TraceContextPropagator::new()); // Propagate context across services

    Ok(tracer)
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let tracer = init_tracer()?;

    // Setup tracing subscriber
    let subscriber = tracing_subscriber::registry()
        .with(tracing_subscriber::fmt::layer())
        .with(tracing_opentelemetry::layer().with_tracer_attributes(vec![
            // Add custom attributes to spans
            opentelemetry::sdk::trace::config::Config().with_sampler(
                opentelemetry::sdk::trace::Sampler::AlwaysOn // Always sample traces for this example
            ),
        ]));
    tracing::subscriber::set_global_default(subscriber).expect("failed to install global subscriber");

    let my_operation = tracer.start("my_operation");
    tracing::info!("Starting my operation...");

    // Simulate some work
    tokio::time::sleep(tokio::time::Duration::from_millis(100)).await;

    let another_span = tracer.start("another_span");
    tracing::debug!("Doing something else...");
    tokio::time::sleep(tokio::time::Duration::from_millis(50)).await;
    another_span.end();

    tracing::info!("My operation finished.");
    my_operation.end();

    Ok(())
}

This code sets up a global tracer, a propagator for context, and a tracing subscriber that bridges tracing events to OpenTelemetry spans. When you run this, you’ll see output like this on your console:

{
  "name": "my_operation",
  "trace_id": "...",
  "span_id": "...",
  "start_time": "...",
  "end_time": "...",
  "parent_span_id": null,
  "attributes": [
    {"key": "sampler.decision", "value": {"type": "boolean", "value": true}}
  ]
}
{
  "name": "another_span",
  "trace_id": "...",
  "span_id": "...",
  "start_time": "...",
  "end_time": "...",
  "parent_span_id": "...",
  "attributes": [
    {"key": "sampler.decision", "value": {"type": "boolean", "value": true}}
  ]
}

The trace_id links all spans belonging to the same request, and parent_span_id shows the causal relationship between spans. This forms a trace, a complete end-to-end view of a request’s journey.

The core problem OpenTelemetry solves is the "distributed monolith" scenario where services are tightly coupled but deployed independently. Without tracing, debugging a slow request that touches multiple services is a nightmare of correlating logs. Tracing provides a unified view.

Internally, OpenTelemetry uses a TracerProvider to manage Tracer instances. Each Tracer creates Spans. These spans have a trace_id (unique to the entire trace) and a span_id (unique to that specific operation within the trace). When one service calls another, it injects the current trace context (including trace_id and span_id) into the request headers using a TextMapPropagator. The receiving service extracts this context and uses it to create new spans that are children of the incoming span, linking the entire operation.

The tracing-opentelemetry crate is the bridge. It hooks into the tracing crate’s event system. When tracing::info!, tracing::span!, or other macros are used, tracing-opentelemetry converts these into OpenTelemetry spans. You can configure which attributes from tracing get attached to your spans.

The Sampler is crucial. For high-volume services, tracing every single request can be too much. Samplers decide whether to keep a trace (e.g., Sampler::AlwaysOn, Sampler::AlwaysOff, or a TraceIdRatioBased sampler). The sampler.decision attribute on the span indicates if it was sampled.

A common pitfall is forgetting to propagate the trace context across service boundaries. If you make an HTTP call from service A to service B, you must use the TraceContextPropagator to inject the current span’s context into the outgoing request headers. Service B then needs to extract this context from the incoming request headers to correctly link its spans.

If you only set up a ConsoleExporter and don’t have a SpanProcessor (like BatchSpanProcessor which with_batch_exporter implicitly uses), spans might be dropped if they are created and ended very quickly, before the exporter has a chance to process them. The Tokio runtime is necessary for the batch exporter to function correctly.

The next concept you’ll likely encounter is setting up a proper backend like Jaeger or Prometheus, and configuring your exporter to send traces there instead of just the console.

Want structured learning?

Take the full Rust course →