Prometheus TLS and Auth: Secure Metrics Endpoints
The most surprising thing about securing Prometheus is how easily you can accidentally expose sensitive operational data to the world, even with TLS enabled.
Let’s see Prometheus in action, scraping a simple HTTP endpoint. We’ll simulate a target service that exposes metrics on port 8080.
# On a mock target service (e.g., a simple Python Flask app)
python -m http.server 8080 &
# On the Prometheus server, with no TLS/auth configured
# prometheus.yml
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'mock-service'
static_configs:
- targets: ['localhost:8080']
When Prometheus scrapes http://localhost:8080/metrics, it’s just plain HTTP. Anyone who can reach that port can see your metrics. This is where TLS and authentication come in.
The Problem: Sensitive Operational Data
Your Prometheus metrics often reveal a lot about your system’s health, performance, and even usage patterns. This can include:
- Resource Utilization: CPU, memory, disk I/O, network traffic.
- Application Performance: Request latency, error rates, throughput.
- User Activity: While not directly PII, aggregated usage can be sensitive.
- Internal System State: Details about your microservices’ communication and health.
Exposing this data unencrypted or without authentication is like leaving your server room door wide open.
The Solution: TLS and Basic Auth
Prometheus supports TLS for encrypting the scrape traffic and Basic Authentication for verifying the identity of the scraping client.
1. Enabling TLS for Scrapes
To enable TLS, you need to configure Prometheus to trust your target’s certificate authority (CA) and provide its own client certificate and key.
On the Target Service (e.g., my-app.com)
You’ll need to configure your web server or application to serve metrics over HTTPS. This involves generating or obtaining TLS certificates. For testing, you can use openssl:
# Generate a self-signed CA
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -sha256 -days 365 -out ca.crt -subj "/CN=MyTestCA"
# Generate a server key and CSR
openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr -subj "/CN=my-app.com"
# Sign the server CSR with your CA
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 365 -sha256
# Configure your web server (e.g., Nginx, Caddy, or a Go app with TLS)
# Example for a Go app:
# http.ListenAndServeTLS(":8443", "server.crt", "server.key", handler)
On the Prometheus Server
You need to tell Prometheus to use TLS and trust the CA that signed your target’s certificate.
ca_file: The path to the CA certificate file.cert_file: The path to the client certificate file Prometheus will use to identify itself.key_file: The path to the client’s private key.server_name: The expectedCNorSANin the target’s certificate.
# prometheus.yml
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'secure-service'
scheme: https # Use HTTPS
static_configs:
- targets: ['my-app.com:8443'] # Target HTTPS port
tls_config:
ca_file: /etc/prometheus/certs/ca.crt
cert_file: /etc/prometheus/certs/prometheus.crt # Client cert for Prometheus
key_file: /etc/prometheus/certs/prometheus.key # Client key for Prometheus
server_name: my-app.com # Matches CN in server.crt
Why it works: Prometheus will now establish a TLS connection to my-app.com:8443. It will verify the server’s certificate against ca.crt and ensure the server’s identity matches my-app.com. The client certificate (prometheus.crt/prometheus.key) is sent to the server for mutual TLS (mTLS) authentication, if the server is configured to require it.
2. Enabling Basic Authentication for Scrapes
If your target requires basic authentication, you can configure Prometheus to send the Authorization: Basic <base64_encoded_username:password> header.
On the Target Service
Your metrics endpoint needs to be protected by HTTP Basic Authentication.
On the Prometheus Server
Add basic_auth_credentials to your scrape configuration.
# prometheus.yml
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'authenticated-service'
scheme: http # Or https if TLS is also used
static_configs:
- targets: ['my-app.com:8080'] # Or :8443 for HTTPS
basic_auth_credentials:
username: 'prometheus_user'
password: 'supersecretpassword'
Why it works: Prometheus will automatically add the Authorization: Basic ... header to its scrape requests, using the provided username and password. The target service validates this header.
3. Combining TLS and Basic Auth
You can, and often should, combine both for maximum security.
# prometheus.yml
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'secure-and-authenticated-service'
scheme: https
static_configs:
- targets: ['my-app.com:8443']
tls_config:
ca_file: /etc/prometheus/certs/ca.crt
cert_file: /etc/prometheus/certs/prometheus.crt
key_file: /etc/prometheus/certs/prometheus.key
server_name: my-app.com
basic_auth_credentials:
username: 'prometheus_user'
password: 'supersecretpassword'
Why it works: Prometheus first establishes a secure TLS connection and then sends the Basic Auth header over that encrypted channel.
The One Thing Most People Don’t Know
Prometheus’s tls_config has a insecure_skip_verify option. While tempting for quick testing, setting this to true completely bypasses certificate validation. This means Prometheus will connect to any server presenting any certificate on the target address, rendering your TLS configuration useless against man-in-the-middle attacks. It’s a shortcut that undermines the entire purpose of TLS.
With these configurations in place, your Prometheus metrics endpoints are protected by encryption and authentication, preventing unauthorized access and ensuring the integrity of your operational data.
The next step is securing the Prometheus server itself, so that only authorized users can access the Prometheus UI and API.