The OpenTelemetry Filter Processor lets you selectively drop spans based on defined rules, preventing metric and trace data bloat from noisy, low-value telemetry.
Let’s see it in action. Imagine you’re getting hammered with traces for health checks hitting your web server. They’re useful for knowing the service is up, but they flood your tracing backend and obscure actual user requests.
Here’s a snippet of an OpenTelemetry Collector configuration using the filter processor to drop these health check spans:
processors:
filter:
traces:
# Drop spans where the name is exactly "GET /health"
span_name:
- pattern: "^GET /health$"
action: drop
# Drop spans from a specific internal client if they're not errors
# This example uses attributes, demonstrating more complex filtering.
# We'll drop spans with 'client.type' attribute equal to 'internal_service_a'
# UNLESS the span's status code is 'ERROR'.
# The 'and' logic is implicit: all conditions must match for the action.
# To express 'OR', you'd need multiple filter blocks.
# This is a simplified example; real-world might involve more attributes.
# For this example, let's assume we want to drop all spans from a specific
# internal service, regardless of their status, to keep it simple.
# Let's refine the example to drop *any* span from our 'internal_service_a'
# if its name *doesn't* contain 'critical'.
# This shows a more nuanced rule: drop if attribute X matches AND attribute Y doesn't.
# Let's simplify again for clarity: drop any span where the 'service.name'
# attribute is 'noisy-service' AND the 'http.status_code' is NOT 500.
# This is a common scenario: dropping successful requests from a known noisy source.
# Let's use a simpler, more direct example for demonstration:
# Drop spans where the 'http.method' attribute is 'OPTIONS'.
# This is common for CORS preflight requests which are often noisy.
expr: 'attributes["http.method"] == "OPTIONS"'
action: drop
receivers:
# ... your receivers (e.g., otlp, jaeger) ...
exporters:
# ... your exporters (e.g., otlp, logging) ...
service:
pipelines:
traces:
receivers: [otlp]
processors: [filter] # Apply the filter processor here
exporters: [logging]
This configuration tells the OpenTelemetry Collector: "If you receive a trace span whose name is exactly 'GET /health', discard it. Also, if a span has the attribute http.method set to OPTIONS, discard that too."
The filter processor works by evaluating rules against individual spans before they are sent to an exporter. These rules can be based on span names, attributes, resource attributes, or even span events. The action can be drop (discard the span) or include (only keep spans that match).
Think of it as a highly configurable bouncer at the door of your tracing backend. It inspects every incoming trace span and decides if it’s worthy of entry based on your rules. You define what "worthy" means.
The core problem this solves is the overwhelming volume of telemetry data generated by modern applications. Services often produce a massive amount of spans, many of which are repetitive, low-value, or simply noise. Health checks, preflight CORS requests, or frequent internal heartbeats can drown out critical user-facing requests, making debugging and performance analysis incredibly difficult and expensive. The filter processor allows you to surgically remove this noise, focusing your resources and attention on what truly matters.
Internally, the filter processor implements a set of rules you define. When a span arrives, it’s passed through these rules sequentially. If a rule matches and its action is drop, the span is immediately discarded and never processed further by subsequent processors or exporters in that pipeline. If the action is include, the span is kept, and other include rules might be evaluated (depending on configuration), but drop rules might still apply if they match. The expr field offers a powerful way to combine multiple conditions using a boolean expression language.
A common misconception is that you can only filter on simple string matches. However, the expr field supports complex boolean logic and comparisons. For instance, you could drop spans from a specific service only if they are not errors and have a duration greater than 1 second: attributes["service.name"] == "my-background-worker" and status.code != "ERROR" and duration_nano > 1000000000. This allows for very granular control.
The next step after filtering noisy spans is often optimizing sampling strategies, as the filter processor operates after spans have been generated and received, not at the source.