How to scrape Prometheus node exporter metrics running in Grafana Alloy

Written by Claudio Kuenzler - 0 comments

Published on January 16th 2025 - Listed in Grafana Observability

---

The node_exporter is the most widely used and known metrics collector (and exporter) in the Prometheus (and surrounding) project. It exposes a lot of system metrics which can be scraped by Prometheus, stored into a TSMD and then used in observability dashboards.

node_exporter runs as a process and (by default) listens on tcp/9100. Besides node_exporter there are many other Prometheus exporters available, all with their own (configurable) listening port. This has an effect on the Prometheus scraping configuration and security (firewall ports). Maybe there's an easier way?

Grafana Alloy: Bundled Prometheus Exporters

Grafana Alloy has an ambitious goal: Be an one-in-all OTLP-compatible observability daemon. It does metrics collection, acts as scraper (of its own collectors) and forwards the metrics to a destination endpoint. It is even capable of being the receiver of such metrics.

Let's focus on the exporters. As of writing this article, Alloy's binary bundles 27 Prometheus exporters, which are named Prometheus components. As one would expect, the famous node_exporter is part of these bundled exporters, is however called "unix" (prometheus.exporter.unix).

Let's use the official documentation example to configure Alloy to run the embedded unix exporter:

ck@linux ~ $ cat /etc/alloy/config.alloy
// Sample config for Alloy.
//
// For a full configuration reference, see https://grafana.com/docs/alloy
logging {
  level = "warn"
}

prometheus.exporter.unix "localhost" {
  include_exporter_metrics = true
}

prometheus.scrape "node" {
  targets = prometheus.exporter.unix.localhost.targets
  forward_to = [
    // although empty, need to define or alloy will not be happy
  ]
}

Attempting to explain this configuration:

• prometheus.exporter.unix "localhost" {...} => This snippet tells Alloy to start the embedded node_exporter, called prometheus.exporter.unix in Alloy. The "localhost" name added at the end is a unique name/label for the node exporter instance. You could run multiple such exporter entries, each with their own label.
  
• prometheus.scrape "default" {...} => Here we tell Alloy to scrape several targets. One of them is the previously configured prometheus.exporter.unix exporter with the label localhost. The ".targets" suffix is necessary so Alloy knows how to talk to the target component. All together (exporter, unique label and targets suffix) results in the following target: prometheus.exporter.unix.localhost.targets .
  
• The forward_to section is mandatory to add - even if there is no forward target - or Alloy will fail to start.
  

According to the Grafana documentation, this should be enough to start collecting the system metrics using the node_exporter a.k.a. prometheus.exporter.unix.

Where are the metrics?

Lets verify if we can access these metrics now:

ck@linux ~ $ curl -s http://localhost:12345/metrics|grep -c node
1

Only one line was returned from Alloy metrics, grepping for the "node" string. Umm, that surely can't be right. 

Comparing with the metrics from a standalone running node_exporter process reveals many more metrics:

ck@linux ~ $ curl -s http://localhost:9100/metrics|grep -c node
2724

Does it mean the Node/Unix exporter metrics can't be scraped or read from Alloy directly?

The component API

As it turns out, each Alloy component exposes its metrics under a very specific path inside the Alloy API. As Alloy is listening on port 12345 (funny, huh?), a curl request to localhost:12345 should reveal the metrics. The tricky part is how to know the API path for each component - as this information can nowhere be found in the documentation!

To access the prometheus.exporter.unix exporter metrics, we need to request the following API URL:

http://localhost:12345/api/v0/component/prometheus.exporter.unix.LABEL/metrics

Where LABEL is obviously our unique label name (localhost), defined in the Alloy configuration. Therefore:

ck@linux ~ $ curl -s localhost:12345/api/v0/component/prometheus.exporter.unix.localhost/metrics |grep -c node
2537

Eureka! We got 2537 lines with the string "node" back!

Comparing with the standalone node_exporter metrics we're more or less in the same range, even though there are less lines. This could also be because of the version of the bundled node_exporter or more comments/descriptions in the output.

So by using the component API path, we are able to locally read the metrics of the Alloy bundled Prometheus exporters; and verify the exporters are working and collecting data.

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.