Prometheus backfilling isn’t about replaying old metrics; it’s about creating new, identical time series that represent the historical data.
Let’s imagine we have a critical service that started emitting metrics two weeks ago, but our Prometheus instance was only set up last week. We want to analyze performance over the last month, but Prometheus only has data for the last week. We need to backfill.
Here’s a common scenario: a new microservice is deployed, and its metrics are immediately available via its /metrics endpoint. But, Prometheus, our primary monitoring system, wasn’t configured to scrape this new service until a few days later. Now, we have a gap in our historical data for this service.
To backfill, we’ll use promtool, Prometheus’s own command-line utility. The core idea is to generate new metric data points for the historical period and then ingest them into Prometheus. We’re not loading old data files; we’re generating new data that looks like it came from the past.
First, we need to get our historical data into a format promtool can understand. Often, this means exporting it from another system or a previous Prometheus instance. Let’s assume we have an export in the Prometheus text format, often called "plain text exposition format." This format looks like this:
# HELP http_requests_total Total number of HTTP requests received.
# TYPE http_requests_total counter
http_requests_total{handler="/api/v1/users",method="POST",status="201"} 100 1678886400
http_requests_total{handler="/api/v1/users",method="POST",status="201"} 105 1678886460
The key here is the timestamp at the end of each line (e.g., 1678886400). This is a Unix timestamp representing seconds since the epoch.
Now, let’s say our historical data is in a file named historical_metrics.prom. We can use promtool to convert this into a format suitable for ingestion. The command promtool tsdb create-blocks is our primary tool. It takes an input directory containing data files and creates an on-disk TSDB block.
If your data is already in the plain text exposition format, you might need to first convert it into a directory structure that promtool expects. A common intermediate step is to create individual files for each metric or time range, though promtool can often work with a single large file if structured correctly.
Let’s assume historical_metrics.prom is our single file. We can create a temporary directory and place our file there.
mkdir temp_data
mv historical_metrics.prom temp_data/
Now, we use promtool to create a TSDB block from this data. The crucial part is specifying the --tsdb.path and the output directory for the blocks.
promtool tsdb create-blocks --tsdb.path=temp_data --output.dir=/path/to/new/blocks
This command reads historical_metrics.prom from temp_data, interprets the timestamps, and creates a new, self-contained TSDB block in /path/to/new/blocks. This block is essentially a snapshot of your historical data, ready to be loaded.
Once you have your block, you need to load it into your running Prometheus instance. This is often done by copying the generated block directory into Prometheus’s data directory and then signaling Prometheus to reload. The exact mechanism depends on your Prometheus setup (systemd, Docker, Kubernetes).
For a typical systemd setup, you’d copy the block directory into /var/lib/prometheus/data/ (or wherever your Prometheus data is stored) and then restart or reload Prometheus:
cp -r /path/to/new/blocks/<block_id> /var/lib/prometheus/data/
systemctl reload prometheus
Prometheus will scan its data directory on startup or reload, discover the new block, and incorporate its data into its queryable time series. The beauty of this is that Prometheus treats these backfilled blocks just like any other data it scraped; there’s no special query syntax needed.
A common pitfall is incorrect timestamp formatting in the input file. Ensure your timestamps are in Unix epoch seconds. If you have milliseconds, you’ll need to divide by 1000. Another issue is metric naming or label inconsistencies between the historical data and what Prometheus expects; promtool is strict about format.
When you first ingest a block, Prometheus might re-evaluate compaction strategies. If you backfill a very large amount of data, it might take some time for the data to become fully queryable as compaction runs.
The next thing you’ll encounter is understanding how these backfilled blocks interact with your existing data during queries, especially regarding rule evaluation and deduplication.