Articles

Prometheus Long-Term Storage: Thanos Setup Guide (2026)

April 18, 2026 6 min read

Thanos makes Prometheus storage look like a child’s toy, by letting you query all your Prometheus metrics across all your clusters, forever.

Let’s see Thanos in action. Imagine you have two Prometheus instances, one in us-east-1 and another in eu-west-2. Normally, to query metrics from both, you’d have to set up some complex federation or remote-write, and even then, you’d likely hit storage limits quickly. Thanos changes this.

Here’s a common setup:

# prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['<node_exporter_ip>:9100']

# This is where Thanos comes in. We'll use the sidecar.
# The sidecar will upload blocks to object storage.
# The query component will then query across these blocks.

Now, the Thanos sidecar. This little guy runs alongside your Prometheus instance. Its primary job is to take the data Prometheus is writing locally and upload it to a long-term storage backend, typically an object store like AWS S3, Google Cloud Storage, or an S3-compatible API.

# thanos-sidecar.yml
 Thanos:
  objstore.config:
    type: s3
    config:
      bucket: "my-thanos-bucket"
      endpoint: "s3.amazonaws.com"
      region: "us-east-1"
      access_key: "AKIA..."
      secret_key: "..."

The Thanos sidecar, when started with the above configuration, will begin uploading Prometheus’s data blocks to the specified S3 bucket. It does this by tailing Prometheus’s WAL (Write Ahead Log) and periodically rolling up data into compact blocks that are then uploaded. This means your Prometheus can keep its local storage small (e.g., just a few hours or days) while Thanos handles the long-term retention.

The magic really happens with the Thanos Query component. This is the entry point for your Grafana or direct promtool queries. It doesn’t store data itself; instead, it discovers and queries data from multiple sources. These sources include:

  1. Thanos Sidecars: Directly querying Prometheus instances that have sidecars running.
  2. Thanos Store Gateways: Querying data that has already been uploaded to object storage (like our S3 bucket).
# thanos-query.yml
 Thanos:
  query:
    replica-label: "replica" # If you have multiple Thanos Query instances for HA
    # This configures where Thanos Query can find data.
    # It will discover sidecars and store gateways automatically.
    # We usually don't need to explicitly list sources here for basic setups.
    # The discovery mechanism is key.

  # This is crucial for querying data from object storage.
  # The Store Gateway fetches data from the object store.
  objstore.config:
    type: s3
    config:
      bucket: "my-thanos-bucket"
      endpoint: "s3.amazonaws.com"
      region: "us-east-1"
      access_key: "AKIA..."
      secret_key: "..."

When you send a query to Thanos Query (e.g., sum(node_cpu_seconds_total) by (instance)), it will:

  1. Contact all available Thanos Sidecars and ask them for recent data (data that hasn’t been uploaded to object storage yet).
  2. Contact all available Thanos Store Gateways. The Store Gateway, using the objstore.config, will list the available data blocks in the object storage (S3 in our case) and query them.
  3. Aggregate the results from all these sources.
  4. Return the final result to you.

This means you can query metrics from Prometheus instances that might be down, or query data from months ago that’s only stored in S3, all through a single query endpoint. Grafana integrates seamlessly by pointing its Prometheus data source to the Thanos Query URL.

The problem Thanos solves is the ephemeral nature of Prometheus’s default storage. Prometheus, by default, stores data locally on disk and has a configurable retention period (e.g., 15 days). Once data is older than this, it’s discarded. Thanos’s sidecar and store gateway architecture decouples storage from the Prometheus scraping process. The sidecar uploads data to durable, scalable object storage, and the store gateway makes this data queryable.

The most surprising thing about Thanos is how it handles data deduplication across multiple Prometheus instances. If you have two Prometheus instances scraping the same targets (common in highly available setups, where each Prometheus might have a replica label), Thanos Query automatically detects and eliminates duplicate data points before returning results, ensuring your aggregations are accurate.

Here’s a glimpse of a docker-compose.yml for a minimal Thanos setup:

version: '3.7'
services:
  prometheus-us-east-1:
    image: prom/prometheus:v2.37.0
    volumes:
      - ./prometheus-us-east-1:/etc/prometheus
      - prometheus_data_us_east_1:/prometheus
    ports:
      - "9090:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=24h' # Keep local data for 24 hours

  thanos-sidecar-us-east-1:
    image: quay.io/thanos/thanos:v0.29.0
    ports:
      - "10901:10901" # gRPC API for sidecar
      - "10902:10902" # HTTP API for metrics
    volumes:
      - ./thanos-sidecar-us-east-1:/etc/thanos
    command:
      - sidecar
      - --tsdb.path=/prometheus # Points to the Prometheus TSDB data directory
      - --objstore.config-file=/etc/thanos/objstore.yml
      - --grpc-address=0.0.0.0:10901
      - --http-address=0.0.0.0:10902
      - --label=region=us-east-1 # Add a label to identify this sidecar's source
      - --receive.local-endpoint=<prometheus_host_ip>:10901 # If using receive for HA

  thanos-store-gateway-us-east-1:
    image: quay.io/thanos/thanos:v0.29.0
    ports:
      - "10100:10100" # HTTP API for store gateway
    volumes:
      - ./thanos-store-gateway-us-east-1:/etc/thanos
    command:
      - store
      - --objstore.config-file=/etc/thanos/objstore.yml
      - --grpc-address=0.0.0.0:10901 # Not strictly needed for store, but common
      - --http-address=0.0.0.0:10100
      - --label=region=us-east-1

  thanos-query:
    image: quay.io/thanos/thanos:v0.29.0
    ports:
      - "9090:9090" # This is the main query endpoint users/Grafana hit
    volumes:
      - ./thanos-query:/etc/thanos
    command:
      - query
      - --http-address=0.0.0.0:9090
      # Thanos Query automatically discovers sidecars via gRPC and store gateways via HTTP/gRPC
      # We configure the object storage for the store gateway part of query
      - --objstore.config-file=/etc/thanos/objstore.yml
      - --label=replica # Deduplication label

volumes:
  prometheus_data_us_east_1:

You’d have a similar set for eu-west-2. The Thanos Query component, when configured with the objstore.yml, will discover the thanos-store-gateway-us-east-1 (via its exposed HTTP API) and thanos-store-gateway-eu-west-2, and also directly connect to the thanos-sidecar-us-east-1 and thanos-sidecar-eu-west-2 via their gRPC APIs.

The most powerful feature, and often overlooked, is Thanos’s ability to perform downsampling. When data is uploaded to object storage, the sidecar can be configured to create lower-resolution versions of the data. This means that queries for older data (e.g., over a year old) will automatically hit the downsampled blocks, dramatically reducing query latency and cost without significant data fidelity loss for long-term trends.

The next thing you’ll want to tackle is setting up Thanos Receive for highly available Prometheus instances, which allows multiple Prometheus instances to write to the same Thanos bucket without a single point of failure.

Want structured learning?

Take the full Prometheus course →