Fix Prometheus Duplicate Sample for Timestamp Error (2026)

Prometheus is failing because the prometheus process is receiving duplicate samples for a given metric with the same timestamp, which is not allowed.

Common Causes and Fixes:

1. Scraping Configuration Errors:

   • Diagnosis: Check your Prometheus scrape configuration (prometheus.yml). Specifically, look for relabel_configs or metric_relabel_configs that might be inadvertently creating duplicate labels or altering timestamps in a way that leads to identical (metric_name, labels, timestamp) tuples. A common mistake is having multiple scrape_configs entries that target the same set of targets with slightly different configurations, leading to duplicate scrapes.
   • Fix: Review your prometheus.yml for redundant or overlapping scrape configurations. Ensure each target is scraped uniquely. If you are using relabel_configs to add or modify labels, carefully inspect them for unintended side effects. For example, if you have two job entries that end up scraping the same set of endpoints and applying different job labels, Prometheus will see them as distinct but the underlying metric data might collide. Consolidate or refine your scrape_configs to ensure each target is uniquely identified and scraped.
   • Why it works: Prometheus identifies unique time series by the combination of metric name and all its labels. If two different scrape configurations target the same physical endpoint and produce metrics with identical names and labels, but arrive at Prometheus with different internal job or instance labels due to configuration, Prometheus might still view them as duplicates if the honor_labels directive is not used correctly or if label modifications lead to identical final label sets. Ensuring unique scrape configurations prevents this.

2. Stale Targets Being Scraped:

   • Diagnosis: Prometheus might be attempting to scrape targets that have recently been removed or are in a transitional state, leading to intermittent duplicate data. Check the Prometheus UI’s "Targets" page for any targets that are oscillating between UP and DOWN or are showing UNKNOWN states.
   • Fix: Ensure your service discovery mechanism is robust and that Prometheus is not attempting to scrape targets that have been deregistered. If using Kubernetes, verify that pods or endpoints are correctly removed from the service discovery when they are terminated. For dynamic environments, ensure a short enough scrape_interval and a reasonable scrape_timeout to quickly identify and stop scraping dead targets.
   • Why it works: If a target is removed but Prometheus still has its address cached or service discovery hasn’t fully propagated the removal, it might attempt a scrape that results in duplicate data being sent before the target is marked as DOWN.

3. Exporter Bugs or Misconfiguration:

   • Diagnosis: The problem might not be with Prometheus itself, but with the exporters (e.g., node_exporter, redis_exporter) that are generating the metrics. Some exporters, especially custom ones or older versions, might have bugs that cause them to emit duplicate samples or corrupt data. Check the logs of your exporters for any errors or unusual behavior.
   • Fix: Update your exporters to their latest stable versions. If you’re using custom exporters, review their code for potential race conditions or incorrect metric handling. For known bugs, apply specific patches or workarounds recommended by the exporter’s maintainers. Ensure the exporter is not being run multiple times on the same instance, which could lead to it scraping its own metrics and reporting them as if they were from a target.
   • Why it works: Exporters are responsible for generating the raw metric data. If an exporter generates the same metric with the same timestamp multiple times within a single scrape, or if multiple instances of the same exporter are running and scraping the same underlying data, Prometheus will receive duplicates.

4. Network Issues Causing Re-Scrapes:

   • Diagnosis: Network instability can sometimes cause Prometheus to lose track of a scrape, leading it to re-initiate the scrape and receive the same data twice. Examine Prometheus’s internal logs (/var/log/prometheus/prometheus.log or similar) for patterns of repeated scrape attempts for the same target within a short period.
   • Fix: Investigate network connectivity between Prometheus and its targets. Ensure there are no packet losses, high latency, or intermittent connectivity issues. Check firewall rules, load balancer configurations, and network infrastructure for potential bottlenecks or misconfigurations that could disrupt scrape requests.
   • Why it works: If a scrape request is sent, Prometheus might mark it as in progress. If the response is delayed or lost due to network issues, Prometheus might time out and initiate a new scrape. If the original response then arrives, or if the target re-sends data, duplicates can occur.

5. Timestamp Skew Between Prometheus and Targets:

   • Diagnosis: While Prometheus is generally tolerant of minor timestamp skew, significant differences between the clock on the Prometheus server and the clocks on the target machines can sometimes manifest as duplicate sample errors, especially if the scrape interval is very short. Use ntpdate or chronyc sources on both Prometheus and target machines to check for clock drift.
   • Fix: Ensure all servers in your environment, including Prometheus and its targets, are synchronized using NTP (Network Time Protocol). Configure NTP clients on all machines to connect to reliable NTP servers and monitor clock drift.
   • Why it works: Prometheus uses the timestamp of a sample to determine if it’s newer than what it already has. If clocks are significantly out of sync, Prometheus might receive an older sample for a metric it has already processed with a newer timestamp, leading to it discarding the older one. However, in some edge cases, if two samples for the exact same metric and labels arrive with identical timestamps (due to clock synchronization issues or the way timestamps are generated), it can trigger the duplicate sample error.

6. Custom Metric Registration Logic:

   • Diagnosis: If you are using client libraries to expose custom metrics from your applications, there might be an issue in how metrics are registered or updated. For example, if a metric is registered multiple times with the same name and labels, or if Set() or Inc() methods are called in a loop without proper de-duplication logic, it can lead to duplicate samples being sent.
   • Fix: Review the code that generates and exposes your custom metrics. Ensure metrics are registered only once. If you are using counters or gauges, ensure that updates are atomic or protected against concurrent access that might lead to re-setting the same value multiple times. Use prometheus.Register carefully and check if a metric already exists before registering.
   • Why it works: Client libraries expect metrics to be registered uniquely. If the same metric is registered multiple times, or if the underlying storage for a metric is updated in a way that generates identical sample points for Prometheus, duplicates can arise.

The next error you might encounter if this is not resolved is Prometheus failing to start or becoming unresponsive due to the persistent ingestion errors.