Prometheus can only discover and scrape metrics from pods if those pods are explicitly annotated to tell Prometheus what to do.
Let’s see Prometheus in action scraping metrics from a Kubernetes pod.
Imagine we have a simple nginx deployment running in Kubernetes. By default, Prometheus won’t know about it. To make Prometheus aware, we need to add specific annotations to the pod’s template within our deployment manifest.
Here’s a typical nginx deployment manifest with the necessary Prometheus annotations:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
annotations:
prometheus.io/scrape: "true"
prometheus.io/path: /metrics
prometheus.io/port: "80"
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
When this deployment is applied, Kubernetes injects these annotations into the pods. Prometheus, configured with the prometheus-operator or a similar discovery mechanism, will then scan for pods matching its kubernetes_sd_config and look for these specific annotations.
The prometheus.io/scrape: "true" annotation is the primary signal. It tells Prometheus, "Yes, this pod is a target for scraping." Without it, Prometheus will ignore the pod entirely, even if it’s exposing metrics on a standard port.
The prometheus.io/path: "/metrics" annotation specifies the HTTP path where Prometheus should retrieve the metrics. For many applications, this is /metrics, but some might expose them on a different path, like /healthz or a custom endpoint. If this annotation is missing, Prometheus defaults to using /metrics.
The prometheus.io/port: "80" annotation indicates the port on the pod that Prometheus should connect to. This is crucial if your application doesn’t expose metrics on the same port as its main service, or if you have multiple ports and need to be explicit. If this is omitted, Prometheus will attempt to scrape on all ports exposed by the container.
With these annotations in place, Prometheus’s Kubernetes service discovery mechanism will identify the nginx-deployment pods. It will then construct a scrape job targeting http://<pod-ip>:<port><path>, for instance, http://10.244.1.5:80/metrics.
The problem this solves is how Prometheus, running outside the pods, can dynamically find and collect metrics from ephemeral, constantly changing pods within a Kubernetes cluster. Kubernetes service discovery, combined with these annotations, provides a declarative way to manage what Prometheus should monitor.
Internally, Prometheus uses Kubernetes API endpoints to get lists of pods. It then filters these pods based on labels and annotations. For pods marked with prometheus.io/scrape: "true", Prometheus creates a target, populating the scrape URL using the pod’s IP address, the port specified in prometheus.io/port, and the path from prometheus.io/path.
The most surprising thing most people don’t realize is that prometheus.io/port can be a named port defined in the Pod’s ports section, not just a numerical value. For example, if your container port is defined as:
ports:
- name: metrics-port
containerPort: 9090
You can then annotate: prometheus.io/port: "metrics-port". This is incredibly useful for avoiding hardcoding port numbers and making your configurations more robust to changes.
Once Prometheus is correctly configured and your pods are annotated, the next step is often to define alerting rules based on the metrics you’re collecting.