OpenTelemetry is a single set of APIs, SDKs, and tools you can use to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) from your applications. It’s designed to be vendor-neutral and extensible.
Let’s see how this works in practice with a simple Python application. Imagine we have a web service that needs to call another internal service. We want to trace the request as it flows through these services.
First, we need to install the necessary OpenTelemetry packages:
pip install opentelemetry-api opentelemetry-sdk opentelemetry-instrumentation-requests opentelemetry-exporter-otlp
Now, let’s set up a basic FastAPI application. We’ll use requests to call another dummy endpoint.
from fastapi import FastAPI
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.requests import RequestsInstrumentor
import requests
import os
# Initialize TracerProvider
provider = TracerProvider()
trace.set_tracer_provider(provider)
# Configure OTLP exporter
# For local testing, you might run an OTLP collector like Jaeger or Tempo
# If your collector is running on localhost:4317
otlp_exporter = OTLPSpanExporter(endpoint="localhost:4317", insecure=True)
span_processor = BatchSpanProcessor(otlp_exporter)
provider.add_span_processor(span_processor)
# Instrument the 'requests' library
RequestsInstrumentor().instrument()
app = FastAPI()
tracer = trace.get_tracer(__name__)
@app.get("/service_a")
async def call_service_b():
with tracer.start_as_current_span("service_a_operation"):
response = requests.get("http://localhost:8001/service_b")
return {"message": "Called Service B", "service_b_response": response.json()}
@app.get("/service_b")
async def service_b_endpoint():
with tracer.start_as_current_span("service_b_operation"):
# Simulate some work
import time
time.sleep(0.1)
return {"message": "Response from Service B"}
# To run this, you'll need uvicorn:
# pip install uvicorn fastapi
# Run this service: uvicorn main:app --port 8000
And a second FastAPI application for service_b:
from fastapi import FastAPI
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
import os
# Initialize TracerProvider
provider = TracerProvider()
trace.set_tracer_provider(provider)
# Configure OTLP exporter
otlp_exporter = OTLPSpanExporter(endpoint="localhost:4317", insecure=True)
span_processor = BatchSpanProcessor(otlp_exporter)
provider.add_span_processor(span_processor)
app = FastAPI()
tracer = trace.get_tracer(__name__)
@app.get("/service_b")
async def service_b_endpoint():
with tracer.start_as_current_span("service_b_operation"):
# Simulate some work
import time
time.sleep(0.1)
return {"message": "Response from Service B"}
# To run this, you'll need uvicorn:
# pip install uvicorn fastapi
# Run this service: uvicorn main_b:app --port 8001
When you run both services and then make a request to /service_a on the first service (e.g., curl http://localhost:8000/service_a), you’ll see traces appear in your configured OTLP collector (like Jaeger or Tempo). You’ll observe a parent span for service_a_operation and a child span for service_b_operation, demonstrating the distributed trace. The RequestsInstrumentor automatically injects trace context into outgoing HTTP requests, allowing service_b to know it’s part of a larger trace.
The core of distributed tracing with OpenTelemetry lies in the concept of SpanContext. When a request enters an instrumented service, if it carries trace context (e.g., from an incoming HTTP header), that context is propagated. If not, a new trace is initiated. This SpanContext contains the trace ID and the parent span ID. When requests makes an outgoing call, the RequestsInstrumentor reads the current SpanContext and injects it into the outgoing HTTP request headers (typically using the W3C Trace-Context standard headers like traceparent and tracestate). The receiving service then extracts this context, creating a new span as a child of the incoming parent span.
A common pitfall is forgetting to set the TracerProvider. Without trace.set_tracer_provider(provider), your spans won’t be processed. Another is misconfiguring the OTLPSpanExporter endpoint; if your collector isn’t running or is on a different port, traces won’t be sent. Forgetting to instrument() the libraries you use (like requests) means those specific operations won’t be automatically traced.
The span processor is where the magic of exporting happens. BatchSpanProcessor is common for production as it buffers spans and sends them in batches, reducing overhead. You can also use SimpleSpanProcessor for debugging, which exports spans immediately. The OTLPSpanExporter is the standard way to send data to OTLP-compatible backends.
You can manually create spans using tracer.start_span("my_operation") and manage their lifecycle with with. This is crucial for instrumenting custom logic or code that isn’t automatically covered by the built-in instrumentation.
The most surprising thing is how seamlessly context propagation happens with libraries like requests once instrumentation is applied. It feels like magic, but it’s a well-defined protocol for injecting and extracting trace identifiers.
The next step is exploring more advanced instrumentation, like custom middleware for frameworks or instrumenting asynchronous operations.