Synthetic Monitoring: Beyond Uptime Pings

Prometheus Blackbox Prober doesn’t actually probe anything itself; it’s just a facade that orchestrates external probes like blackbox_exporter and reports their results back to Prometheus.

Let’s see this in action. Imagine you have a web service running at http://my-awesome-app.example.com. You want to ensure it’s up and responding correctly.

First, you need to deploy the blackbox_exporter. This is a standalone service that understands how to perform various network probes (HTTP, TCP, ICMP, gRPC, DNS). Here’s a minimal blackbox.yml configuration for HTTP probes:

modules:
  http_2xx:
    prober: http
    http:
      method: GET
      # Successfully probe if the status code is between 200 and 299
      valid_status_codes: [200, 201, 202, 203, 204, 205, 206, 207, 208, 226]
      # Optional: Specify a body that must be present in the response
      # fail_if_body_not_contains: "Welcome to My Awesome App!"
      # Optional: Specify a body that must NOT be present in the response
      # fail_if_body_contains: "Maintenance Mode"
  http_post_2xx:
    prober: http
    http:
      method: POST
      # You can also specify request body and headers
      # body: '{"key": "value"}'
      # headers:
      #   Content-Type: application/json
      valid_status_codes: [200]

You’d run the blackbox_exporter with this config, exposing it on a port, say 9115:

docker run --rm -p 9115:9115 -v $(pwd)/blackbox.yml:/config/blackbox.yml prom/blackbox-exporter:latest --config.file=/config/blackbox.yml

Now, your Prometheus server needs to be configured to scrape the blackbox_exporter, but not directly. Instead, you’ll use Prometheus’s relabel_configs to tell Prometheus to send probe requests to the blackbox_exporter for your target application.

In your prometheus.yml (or a separate scrape config file), you’ll have a job like this:

scrape_configs:
  - job_name: 'blackbox_http'
    metrics_path: /probe
    params:
      module: [http_2xx] # This refers to the module defined in blackbox.yml
    static_configs:
      - targets:
        - http://my-awesome-app.example.com # The actual target to probe
        - https://another-service.example.com
    relabel_configs:
      # This is the magic: Prometheus will scrape the blackbox_exporter,
      # but the labels it uses will be based on the *original* target.
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter.example.com:9115 # Address of your blackbox_exporter

Here’s what’s happening:

1. Prometheus sees http://my-awesome-app.example.com as a target.
2. It applies the relabel_configs. The first rule takes the __address__ (which is http://my-awesome-app.example.com) and puts it into a parameter named target (__param_target). This parameter is sent to the blackbox_exporter when Prometheus scrapes it.
3. The metrics_path is /probe, which is the endpoint blackbox_exporter listens on for probe requests.
4. The __address__ is then rewritten to point to your blackbox_exporter’s address (blackbox-exporter.example.com:9115).
5. So, Prometheus scrapes http://blackbox-exporter.example.com:9115/probe?target=http://my-awesome-app.example.com&module=http_2xx.
6. The blackbox_exporter receives this request, reads the target and module parameters, performs an HTTP GET against http://my-awesome-app.example.com using the http_2xx module’s configuration, and returns metrics like probe_success (0 or 1), probe_duration_seconds, probe_http_status_code, etc., back to Prometheus.
7. Prometheus then stores these metrics, with the instance label correctly set to http://my-awesome-app.example.com.

The most surprising thing about this setup is that Prometheus itself doesn’t know how the probe is being done; it just knows it’s scraping an endpoint that reports probe results. The intelligence of how to probe (HTTP, TCP, DNS, etc.) resides entirely within the blackbox_exporter. You can even run multiple blackbox_exporter instances and have Prometheus route probes to different ones based on labels, allowing for geographically distributed probing.

The key to making this work is understanding that the relabel_configs are not just for filtering targets, but for transforming the scrape request itself. You’re telling Prometheus to scrape blackbox_exporter, but to pass specific parameters (__param_target, __param_module) that instruct blackbox_exporter on what to probe and how.

When you’re setting up alerts, you’ll typically be alerting on probe_success == 0.

The next thing you’ll likely encounter is wanting to probe internal services that aren’t publicly accessible, leading you to explore service discovery integration with Blackbox Prober.