Prometheus doesn’t actually scrape data at a fixed interval; it evaluates its scrape configuration and initiates scrapes based on that.

Let’s see what this looks like in practice. Imagine you have a prometheus.yml configuration like this:

scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']
    scrape_interval: 30s
    evaluation_interval: 15s

When Prometheus starts, it reads this configuration. The scrape_interval: 30s tells Prometheus that after it successfully scrapes localhost:9100, it should wait 30 seconds before initiating the next scrape for that target. It doesn’t mean "at exactly 0, 30, 60 seconds past the minute." It’s a relative delay.

The evaluation_interval: 15s is where things get interesting. This interval dictates how often Prometheus evaluates its alerting and recording rules. For each rule, Prometheus checks if the condition defined in the rule is met based on the current time series data. If a recording rule’s condition is met, Prometheus records a new time series based on the rule’s expression. If an alerting rule’s condition is met, Prometheus marks the alert as pending.

Here’s a common misconception: many assume scrape_interval and evaluation_interval are directly tied, or that evaluation_interval must be a multiple of scrape_interval. This isn’t true. Prometheus independently manages these two timers.

Let’s trace a few cycles with the config above:

  1. Time 0s: Prometheus starts. It reads the config. It sees job_name: 'node_exporter' with scrape_interval: 30s and evaluation_interval: 15s.
  2. Time 0.1s (approx): Prometheus initiates the first scrape for localhost:9100.
  3. Time 0.5s (approx): The scrape for localhost:9100 completes, returning metrics. These metrics are stored in Prometheus’s internal time series database.
  4. Time 15s: The evaluation_interval triggers. Prometheus evaluates all recording and alerting rules. It checks the metrics scraped at 0.5s against the rule conditions.
  5. Time 30s: The scrape_interval timer, which started counting after the scrape finished at 0.5s, expires. Prometheus initiates the second scrape for localhost:9100.
  6. Time 30.4s (approx): The second scrape completes, returning new metrics.
  7. Time 45s: The evaluation_interval triggers again. Prometheus evaluates rules against the metrics scraped at 0.5s and 30.4s.
  8. Time 60s: The scrape_interval timer, which started counting after the scrape finished at 30.4s, expires. Prometheus initiates the third scrape.

Notice that evaluations happen more frequently than scrapes. This means Prometheus can react to changes in your metrics (via alerts or recording rules) much faster than it updates the raw data itself.

The actual mechanism is that Prometheus maintains a next_scrape timestamp for each target and a next_evaluation timestamp for its rules. When a scrape is completed, next_scrape is set to current_time + scrape_interval. When an evaluation cycle completes, next_evaluation is set to current_time + evaluation_interval. A background worker continuously checks these timestamps and triggers actions when they are due.

The critical insight is that scrape_interval controls how fresh your raw metric data is, while evaluation_interval controls how fresh your derived data (alerts and recorded metrics) is. Setting evaluation_interval to be much smaller than scrape_interval allows for rapid alerting on potentially transient metric changes, even if Prometheus isn’t fetching the absolute latest data points from the target itself. This is a performance optimization: evaluating rules is generally much cheaper than scraping targets, especially if targets are numerous or slow to respond.

The most surprising thing is that if a scrape takes longer than the scrape_interval, Prometheus will not start a new scrape for that target until the current one finishes and the scrape_interval has elapsed after the completion. This can lead to targets being scraped less frequently than configured if they are consistently slow.

The next concept you’ll likely encounter is how Prometheus handles multiple targets within a single job, and how scrape and evaluation intervals apply at the target level, not just the job level.

Want structured learning?

Take the full Prometheus course →