The OpenTelemetry Kubernetes Operator can automatically inject instrumentation into your application pods without you needing to modify your application code.
Imagine you have a Kubernetes cluster, and you want to get telemetry data (like traces, metrics, and logs) from your applications running in pods. Traditionally, you’d have to manually add OpenTelemetry SDKs to each application, configure them, and redeploy. The Operator automates this by modifying the pod’s container definition before it starts.
Here’s how it works in practice. Let’s say you have a simple Deployment for a web application:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-web-app
spec:
replicas: 2
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: web-container
image: my-repo/my-web-app:latest
ports:
- containerPort: 8080
To enable auto-instrumentation, you’d create an Instrumentation resource. This resource tells the Operator how to instrument.
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: my-instrumentation
spec:
dotnet:
image: ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-dotnet-instrumentation:v0.3.0
java:
image: ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-java-instrumentation:v0.17.0
nodejs:
image: ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-nodejs-instrumentation:v0.33.0
python:
image: ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-python-instrumentation:v0.3.0
# For Go, you typically need to compile it in, or use a sidecar.
# The operator can configure a sidecar for Go.
# For other languages, it injects an agent as a sidecar.
Once this Instrumentation resource is applied, the Operator watches for pods that match specific labels (or you can explicitly target them). When it finds a pod that needs instrumentation, it mutates the pod’s definition. It adds an initContainer that downloads the appropriate language-specific agent and then modifies the main container’s command and args to launch the application through that agent.
For example, if you had a Java application, the Operator might transform your pod spec to look something like this (simplified):
# ... (original pod spec) ...
spec:
initContainers:
- name: opentelemetry-init-container
image: ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-java-instrumentation:v0.17.0
command: ['/javaagent/download-and-extract.sh'] # This script downloads the agent
volumeMounts:
- name: opentelemetry-agent
mountPath: /javaagent
containers:
- name: web-container
image: my-repo/my-web-app:latest
ports:
- containerPort: 8080
env:
- name: JAVA_TOOL_OPTIONS
value: "-javaagent:/javaagent/opentelemetry-javaagent.jar" # This is the key part
# ... (other container specs) ...
volumes:
- name: opentelemetry-agent
emptyDir: {} # Shared volume for init container to place agent
The JAVA_TOOL_OPTIONS environment variable is how the Java Virtual Machine knows to load the OpenTelemetry Java agent. The agent then intercepts your application’s runtime, adds spans for incoming/outgoing requests, database calls, etc., and sends them to a configured collector.
The Instrumentation resource also defines where to send the telemetry. This is done via the collector field, which points to an OpenTelemetry Collector deployment.
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: my-instrumentation
spec:
# ... (language specific config) ...
collector:
name: otel-collector # Name of the OpenTelemetry Collector service
namespace: opentelemetry # Namespace where the Collector is deployed
The Collector is the central hub. It receives telemetry data from your instrumented applications, processes it (e.g., batches, filters, samples), and exports it to your backend observability tools (like Jaeger, Prometheus, Grafana Loki, etc.).
The Operator’s power lies in its ability to dynamically modify pod definitions. It uses Kubernetes Admission Controllers (specifically, a Mutating Webhook Configuration) to intercept pod creation requests. When a pod is about to be created, the webhook is called, and if the pod matches the criteria defined in an Instrumentation resource (e.g., has a specific label, or is in a specific namespace), the Operator injects the necessary sidecars and environment variables. You can target specific deployments or namespaces using selectors in the Instrumentation resource.
Crucially, the Operator handles language-specific details. For Java, it injects JAVA_TOOL_OPTIONS. For Python, it modifies PYTHONPATH and OTEL_PYTHON_INSTRUMENTATION_CONFIGURATOR. For Node.js, it injects NODE_OPTIONS. For .NET, it sets CORECLR_ENABLE_PROFILING and CORECLR_PROFILER. This abstraction means you don’t need to know the exact environment variables or command-line flags for each language; the Operator figures it out based on the Instrumentation resource configuration.
If you’re using Go applications, auto-instrumentation typically involves compiling the OpenTelemetry Go SDK directly into your application. The Operator can’t dynamically inject a Go agent in the same way it does for other languages. However, it can be configured to deploy a Go application as a sidecar to your main application, which can then be used for collecting and exporting telemetry.
The most common way to manage what gets instrumented is by adding a label to your Deployment or Pod’s template.metadata.labels. For example, adding opentelemetry.io/instrumentation: "true" to your pod template. The Instrumentation resource then has a selector field that matches this label.
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: my-instrumentation
spec:
# ...
selector:
matchLabels:
opentelemetry.io/instrumentation: "true"
When you apply this to your Deployment’s pod template:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-web-app
spec:
# ...
template:
metadata:
labels:
app: web
opentelemetry.io/instrumentation: "true" # This label triggers the Operator
spec:
containers:
# ...
The Operator sees this label and injects the instrumentation. If you remove the label, the Operator will eventually stop managing that pod’s instrumentation (though the injected agent might still be running until the pod is recreated).
This mechanism provides a powerful, declarative way to manage application instrumentation across your Kubernetes cluster, decoupling the observability setup from your application development lifecycle.
The next step after successfully auto-instrumenting your pods is often configuring advanced sampling strategies within the OpenTelemetry Collector to manage the volume of telemetry data generated.