Prometheus’s without() and ignoring() clauses are not just syntactic sugar; they fundamentally alter how vector matching works, allowing you to perform set operations on your time series labels.

Let’s see this in action. Imagine you have two metrics: http_requests_total and http_requests_failed_total. Both have labels like method, path, and status.

# http_requests_total:
# {method="GET", path="/", status="200"} 100
# {method="POST", path="/", status="200"} 50
# {method="GET", path="/users", status="200"} 75
# {method="GET", path="/users", status="404"} 10

# http_requests_failed_total:
# {method="GET", path="/", status="500"} 5
# {method="POST", path="/", status="400"} 2
# {method="GET", path="/users", status="404"} 3

Now, let’s try to calculate the success rate for requests ignoring the status label.

sum by (method, path) (http_requests_total)
/
sum by (method, path) (http_requests_total)
  ignoring (status)

This query will produce a success rate by aggregating requests across all status codes for a given method and path. The ignoring (status) clause tells Prometheus to match series based on method and path only, effectively treating all status labels as if they were the same during the join.

What if we wanted to see the failure rate, but only for requests that didn’t have a 200 status code?

sum by (method, path) (http_requests_failed_total)
/
sum by (method, path) (http_requests_total)
  without (status)

Here, without (status) means that the http_requests_total metric will only be considered for matching if its status label is not present in the http_requests_failed_total metric’s labels. In essence, it’s performing a set difference on the label values for status. This query would give us the proportion of failed requests out of the total requests that did not end with a 200 status code (because http_requests_failed_total by definition won’t have status="200").

The core problem these clauses solve is the ambiguity in vector matching when you have multiple common labels. By default, Prometheus requires an exact match on all labels for a vector match to occur. ignoring and without give you fine-grained control over this matching behavior.

Let’s break down the mental model. When Prometheus performs a binary operation (like / or +) between two vectors, it needs to decide which series from the left-hand side vector should be paired with which series from the right-hand side vector.

  1. Default Matching: Prometheus looks for series with identical label sets on both sides. If vector1{a="1", b="2"} exists on the left and vector2{a="1", b="2"} exists on the right, they are matched. If vector1{a="1", b="3"} exists on the left, it won’t find a match on the right if no series has a="1", b="3".

  2. ignoring Clause: vector1 / vector2 ignoring (label_to_ignore) modifies step 1. When matching, Prometheus will ignore the specified label (label_to_ignore) on both sides. So, vector1{a="1", b="2", c="X"} on the left and vector2{a="1", b="2", c="Y"} on the right will be matched because a="1" and b="2" are identical, and c is not considered for matching. The operation is then performed on the matched pairs. If a label is only present on one side, it’s still ignored.

  3. without Clause: vector1 / vector2 without (label_to_exclude) also modifies step 1, but in a more restrictive way. Prometheus will only match series where the labels other than label_to_exclude are identical. Critically, label_to_exclude must not be present in the matched series. If vector1{a="1", b="2", c="X"} is on the left and vector2{a="1", b="2"} is on the right, they will match because a="1" and b="2" are identical, and c is not present on the right-hand side. However, if vector1{a="1", b="2", c="X"} is on the left and vector2{a="1", b="2", c="Y"} is on the right, they will not match because c is present on both sides, and without implies that the label shouldn’t be shared between the matched pair. This is where the "set difference" idea comes from: it’s about excluding pairs where the specified label exists on both sides of the match.

This distinction is crucial: ignoring allows common labels to differ, while without enforces that the specified label cannot be common to both sides of a matched pair.

A common pitfall is assuming without acts like a simple label filter. It doesn’t. It’s a constraint on the vector matching process itself. If you have vector1{app="api", env="prod"} and vector2{app="api", env="staging"}, and you use ignoring(env), they’ll match on app="api". If you use without(env), they won’t match because env is present on both sides.

The next hurdle is understanding how these clauses interact with aggregations (sum, avg, etc.) and the on() clause, which provides explicit control over which labels must match.

Want structured learning?

Take the full Prometheus course →