Prometheus Node Exporter can expose custom metrics by simply dropping a shared object file (.so) into its collector directory.
Let’s see this in action. Imagine you have a legacy application that exposes a single integer metric, myapp_requests_total, on a local TCP port. Node Exporter doesn’t know about this, but we can teach it.
First, we need a C program that implements the Node Exporter collector interface. This interface is pretty straightforward: a _init function to register the collector and a _fini function for cleanup. The core logic lives in a _collect function.
#include <prometheus/collector.h>
#include <prometheus/registry.h>
#include <prometheus/gauge.h>
#include <prometheus/text_collector.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <arpa/inet.h>
// A simple struct to hold our metric data
typedef struct {
prometheus_collector_t collector;
prometheus_metric_t* requests_total;
int port;
} myapp_collector_t;
// Function to fetch the metric from the legacy app
static int fetch_myapp_metric(int port) {
int sockfd = socket(AF_INET, SOCK_STREAM, 0);
if (sockfd < 0) {
perror("socket");
return -1;
}
struct sockaddr_in serv_addr;
memset(&serv_addr, '0', sizeof(serv_addr));
serv_addr.sin_family = AF_INET;
serv_addr.sin_port = htons(port);
if (inet_pton(AF_INET, "127.0.0.1", &serv_addr.sin_addr) <= 0) {
perror("inet_pton");
close(sockfd);
return -1;
}
if (connect(sockfd, (struct sockaddr *)&serv_addr, sizeof(serv_addr)) < 0) {
perror("connect");
close(sockfd);
return -1;
}
char buffer[1024] = {0};
ssize_t bytes_read = read(sockfd, buffer, sizeof(buffer) - 1);
if (bytes_read < 0) {
perror("read");
close(sockfd);
return -1;
}
buffer[bytes_read] = '\0';
close(sockfd);
// Assuming the legacy app returns a simple integer string
return atoi(buffer);
}
// The collect function that Node Exporter will call
static void myapp_collect(prometheus_collector_t* collector, prometheus_registry_t* registry) {
myapp_collector_t* myapp_collector = (myapp_collector_t*)collector;
int value = fetch_myapp_metric(myapp_collector->port);
if (value >= 0) {
prometheus_metric_set_value(myapp_collector->requests_total, (double)value);
} else {
// Handle error, perhaps log it or set a specific error metric
fprintf(stderr, "Failed to fetch myapp metric\n");
}
}
// Initialization function called by Node Exporter
static int myapp_collector_init(prometheus_collector_t* collector, const char* config) {
myapp_collector_t* myapp_collector = (myapp_collector_t*)collector;
// Parse config for port, e.g., "port=9876"
sscanf(config, "port=%d", &myapp_collector->port);
if (myapp_collector->port == 0) {
myapp_collector->port = 9876; // Default port
}
// Register the collector with the registry
prometheus_registry_register_collector(prometheus_collector_registry_get(), collector);
// Create the metric
prometheus_metric_options_t options = {
.name = "myapp_requests_total",
.help = "Total requests to the legacy application",
.type = PROMETHEUS_METRIC_COUNTER, // Or GAUGE if it's not cumulative
.labels = NULL,
.num_labels = 0
};
myapp_collector->requests_total = prometheus_metric_new(&options);
prometheus_collector_add_metric(collector, myapp_collector->requests_total);
return 0;
}
// Cleanup function
static void myapp_collector_fini(prometheus_collector_t* collector) {
// Cleanup resources if any
(void)collector; // Suppress unused variable warning
}
// The structure Node Exporter looks for
prometheus_collector_plugin_t collector_plugin = {
.name = "myapp_collector",
.init = myapp_collector_init,
.collect = myapp_collect,
.fini = myapp_collector_fini,
};
To build this, you’ll need the Node Exporter’s collector development headers. You can usually find these within the Node Exporter source code or by installing a development package if available. Assuming you have them, you’d compile like this:
gcc -shared -fPIC -o myapp_collector.so myapp_collector.c -I/path/to/node_exporter/internal/github.com/prometheus/client_golang/prometheus -L/path/to/node_exporter/internal/github.com/prometheus/client_golang/prometheus -lprometheus_collector
This command compiles myapp_collector.c into a shared object file named myapp_collector.so. The -shared and -fPIC flags are crucial for creating a position-independent code shared library. The -I and -L flags point to the Prometheus client library headers and libraries, and -lprometheus_collector links against the collector library. The exact paths will depend on your Node Exporter installation or build environment.
Once compiled, you place myapp_collector.so into the Node Exporter’s collector directory. This is typically specified via the --collector.directory flag when starting Node Exporter. If you don’t specify it, it defaults to a directory relative to the Node Exporter binary. Let’s assume you’ve set it to /opt/node_exporter/collectors.
sudo cp myapp_collector.so /opt/node_exporter/collectors/
Then, restart or start your Node Exporter instance, ensuring the collector directory is pointed correctly. You might also need to explicitly enable your custom collector if Node Exporter has a mechanism for that (though often, just placing the .so is enough).
/path/to/node_exporter/node_exporter --collector.directory=/opt/node_exporter/collectors/ --web.listen-address=":9100"
Now, if your legacy application is running and exposing 123 on port 9876, you can query Prometheus. Your Node Exporter will fetch this value and expose it as myapp_requests_total.
curl http://localhost:9100/metrics | grep myapp_requests_total
You should see output like:
# HELP myapp_requests_total Total requests to the legacy application
# TYPE myapp_requests_total counter
myapp_requests_total 123
The key insight here is that Node Exporter’s plugin system is designed to be extensible at runtime. It dynamically loads shared libraries that adhere to a specific C API. This allows you to expose virtually any metric from your system, even from applications that don’t natively speak the Prometheus exposition format.
The prometheus_metric_set_value function is the workhorse for updating the metric’s value. It takes a prometheus_metric_t* pointer and the new value as a double. The Node Exporter’s internal registry then handles formatting this into the Prometheus text exposition format when scraped.
A common pitfall is incorrect linking or missing dependencies for the shared object. Ensure your gcc command correctly points to the Prometheus client library headers and libraries. Another is failing to handle network errors gracefully in your fetch_myapp_metric function; Node Exporter expects collectors to be robust and not crash the exporter process.
After fixing your custom collector, the next thing you might encounter is the need to expose multiple metrics from your legacy application, or perhaps metrics with labels.