An OpenTelemetry sidecar intercepts network traffic from an application pod to capture telemetry signals like traces and metrics.
Imagine you have a microservice application running in Kubernetes. Each service is a separate pod, and you want to collect performance data from them without modifying the application code itself. This is where the OpenTelemetry sidecar pattern shines. A small, dedicated container (the sidecar) is deployed alongside your application container within the same pod. This sidecar is configured to receive, process, and export telemetry data generated by the application.
Here’s a simplified look at it in action. Let’s say you have a frontend service and a backend service, both running in Kubernetes.
Pod Definition (frontend-pod.yaml):
apiVersion: v1
kind: Pod
metadata:
name: frontend-app
spec:
containers:
- name: frontend-container
image: my-frontend-app:latest
ports:
- containerPort: 8080
# Application specific configuration
- name: otel-sidecar
image: otel/opentelemetry-collector-contrib:0.90.0
ports:
- containerPort: 8888 # Prometheus metrics
- containerPort: 8081 # OTLP receiver
env:
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
# Sidecar specific configuration will go here
The otel-sidecar container in this pod is running the OpenTelemetry Collector. The key idea is that the frontend-container is configured to send its telemetry data to the sidecar, usually via a local network address within the pod (e.g., localhost:8081 for OTLP). The sidecar then takes this data and forwards it to a central telemetry backend like Prometheus or Jaeger.
How it works internally:
The magic happens through the OpenTelemetry Collector’s configuration within the sidecar. The collector is a vendor-agnostic agent that can receive, process, and export telemetry data.
-
Receivers: The sidecar’s collector is configured with receivers that listen for incoming telemetry data. For OTLP (OpenTelemetry Protocol), this might be
otlpreceiver listening on port8081. For Prometheus metrics scraped from the application, it might be aprometheusreceiver configured to scrape an endpoint exposed by the application container. -
Processors: Once data is received, processors can be applied. This is where you can add attributes (like Kubernetes pod name, namespace, etc.), sample traces, or perform other transformations. For example, a
k8sattributesprocessor can automatically enrich telemetry with Kubernetes metadata. -
Exporters: Finally, exporters send the processed data to your chosen backend systems. This could be an
otlpexporter to send traces to an OpenTelemetry Collector agent or directly to a backend like Jaeger, or aprometheusexporter to expose metrics for Prometheus to scrape.
The problem this solves:
The primary problem this pattern solves is decoupling telemetry collection from application development. Developers can focus on writing application logic, and the operational concerns of collecting and exporting telemetry are handled by the sidecar. This means:
- No code changes: You don’t need to add OpenTelemetry SDKs or modify your application’s HTTP handlers to inject trace context.
- Consistent collection: All applications using the sidecar pattern will have their telemetry collected and exported in a uniform way.
- Simplified deployment: The telemetry collection logic is part of the pod definition, making it easy to roll out and manage.
Example Collector Configuration (otel-collector-config.yaml within the sidecar):
receivers:
otlp:
protocols:
grpc:
http:
prometheus:
config:
scrape_configs:
- job_name: 'frontend-app'
static_configs:
- targets: ['localhost:8080'] # Assuming app exposes metrics on 8080
processors:
k8sattributes:
auth_type: "serviceAccount"
batch:
exporters:
otlp:
endpoint: "otel-collector-agent.observability.svc.cluster.local:4317" # Export OTLP to a central collector
prometheus:
endpoint: "0.0.0.0:8888" # Expose Prometheus metrics for scraping
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp]
metrics:
receivers: [prometheus]
processors: [batch, k8sattributes]
exporters: [prometheus]
In this configuration, the sidecar receives OTLP traces from the frontend-container on localhost:8081, adds Kubernetes attributes to metrics scraped from localhost:8080, and then exports both to a central otel-collector-agent and exposes metrics on port 8888.
The most surprising thing about this pattern is how it allows you to inject powerful, standardized telemetry collection into existing applications with zero code modification, as long as they expose their telemetry on standard ports or protocols that the sidecar can understand. It’s a form of "observability injection" that doesn’t require you to be the author of the original application.
The next step after implementing this is often to ensure that trace context propagation is handled correctly across services, which might involve instrumenting the application code to pass trace headers, or relying on more advanced sidecar techniques for network-level injection.