OpenTelemetry traces reveal unexpected bottlenecks by showing how requests flow through your services, not just within them.
Let’s instrument a simple Rust web service to see this in action. We’ll use tracing for instrumentation and opentelemetry-stdout to export our traces to the console for now.
use opentelemetry::{
global,
trace::{TraceContext},
sdk::propagation::TraceContextPropagator,
trace::TraceError,
};
use opentelemetry_sdk::{
trace::{Tracer, TracerProvider},
export::trace::SpanExporter,
runtime::Tokio,
};
use opentelemetry_stdout::ConsoleExporter;
use tracing::{subscriber::Subscriber, util::SubscriberExt};
use tracing_subscriber::{layer::SubscriberExt, Registry};
fn init_tracer() -> Result<Tracer, TraceError> {
let exporter = ConsoleExporter::default(); // Export to console
let tracer_provider = opentelemetry_sdk::trace::TracerProvider::builder()
.with_simple_exporter(exporter)
.with_batch_exporter(exporter, opentelemetry_sdk::runtime::Tokio) // Use Tokio runtime for batching
.build();
let tracer = tracer_provider.get_tracer("my-rust-app");
global::set_tracer_provider(tracer_provider);
global::set_text_map_propagator(TraceContextPropagator::new()); // Propagate context across services
Ok(tracer)
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let tracer = init_tracer()?;
// Setup tracing subscriber
let subscriber = tracing_subscriber::registry()
.with(tracing_subscriber::fmt::layer())
.with(tracing_opentelemetry::layer().with_tracer_attributes(vec![
// Add custom attributes to spans
opentelemetry::sdk::trace::config::Config().with_sampler(
opentelemetry::sdk::trace::Sampler::AlwaysOn // Always sample traces for this example
),
]));
tracing::subscriber::set_global_default(subscriber).expect("failed to install global subscriber");
let my_operation = tracer.start("my_operation");
tracing::info!("Starting my operation...");
// Simulate some work
tokio::time::sleep(tokio::time::Duration::from_millis(100)).await;
let another_span = tracer.start("another_span");
tracing::debug!("Doing something else...");
tokio::time::sleep(tokio::time::Duration::from_millis(50)).await;
another_span.end();
tracing::info!("My operation finished.");
my_operation.end();
Ok(())
}
This code sets up a global tracer, a propagator for context, and a tracing subscriber that bridges tracing events to OpenTelemetry spans. When you run this, you’ll see output like this on your console:
{
"name": "my_operation",
"trace_id": "...",
"span_id": "...",
"start_time": "...",
"end_time": "...",
"parent_span_id": null,
"attributes": [
{"key": "sampler.decision", "value": {"type": "boolean", "value": true}}
]
}
{
"name": "another_span",
"trace_id": "...",
"span_id": "...",
"start_time": "...",
"end_time": "...",
"parent_span_id": "...",
"attributes": [
{"key": "sampler.decision", "value": {"type": "boolean", "value": true}}
]
}
The trace_id links all spans belonging to the same request, and parent_span_id shows the causal relationship between spans. This forms a trace, a complete end-to-end view of a request’s journey.
The core problem OpenTelemetry solves is the "distributed monolith" scenario where services are tightly coupled but deployed independently. Without tracing, debugging a slow request that touches multiple services is a nightmare of correlating logs. Tracing provides a unified view.
Internally, OpenTelemetry uses a TracerProvider to manage Tracer instances. Each Tracer creates Spans. These spans have a trace_id (unique to the entire trace) and a span_id (unique to that specific operation within the trace). When one service calls another, it injects the current trace context (including trace_id and span_id) into the request headers using a TextMapPropagator. The receiving service extracts this context and uses it to create new spans that are children of the incoming span, linking the entire operation.
The tracing-opentelemetry crate is the bridge. It hooks into the tracing crate’s event system. When tracing::info!, tracing::span!, or other macros are used, tracing-opentelemetry converts these into OpenTelemetry spans. You can configure which attributes from tracing get attached to your spans.
The Sampler is crucial. For high-volume services, tracing every single request can be too much. Samplers decide whether to keep a trace (e.g., Sampler::AlwaysOn, Sampler::AlwaysOff, or a TraceIdRatioBased sampler). The sampler.decision attribute on the span indicates if it was sampled.
A common pitfall is forgetting to propagate the trace context across service boundaries. If you make an HTTP call from service A to service B, you must use the TraceContextPropagator to inject the current span’s context into the outgoing request headers. Service B then needs to extract this context from the incoming request headers to correctly link its spans.
If you only set up a ConsoleExporter and don’t have a SpanProcessor (like BatchSpanProcessor which with_batch_exporter implicitly uses), spans might be dropped if they are created and ended very quickly, before the exporter has a chance to process them. The Tokio runtime is necessary for the batch exporter to function correctly.
The next concept you’ll likely encounter is setting up a proper backend like Jaeger or Prometheus, and configuring your exporter to send traces there instead of just the console.