Prometheus

Prometheus is an open-source event monitoring and alerting application. It records metrics in a time series database built by pulling (also known as scraping) metrics from different services, with flexible queries and real-time alerting.

Prometheus

Metrics aggregation system Application

Status	Active
Language(s)	Go
License	Apache License 2.0
External links
GitHub	prometheus/prometheus
Documentation	https://prometheus.io/docs/introduction/overview/

Prometheus exporters

Prometheus works by scraping from HTTP endpoints, which are often provided by Prometheus exporters.

node_exporter

Below is an example of prometheus node_exporter with additional collectors enabled. node_exporter is documented in the NixOS manual.

❄︎ /etc/nixos/configuration.nix

{ config, pkgs, ... }:
{
  # https://nixos.org/manual/nixos/stable/#module-services-prometheus-exporters
  # https://github.com/NixOS/nixpkgs/blob/nixos-24.05/nixos/modules/services/monitoring/prometheus/exporters.nix
  services.prometheus.exporters.node = {
    enable = true;
    port = 9000;
    # For the list of available collectors, run, depending on your install:
    # - Flake-based: nix run nixpkgs#prometheus-node-exporter -- --help
    # - Classic: nix-shell -p prometheus-node-exporter --run "node_exporter --help"
    enabledCollectors = [
      "ethtool"   
      "softirqs"
      "systemd"
      "tcpstat"
      "wifi"
    ];
    # You can pass extra options to the exporter using `extraFlags`, e.g.
    # to configure collectors or disable those enabled by default.
    # Enabling a collector is also possible using "--collector.[name]",
    # but is otherwise equivalent to using `enabledCollectors` above.
    extraFlags = [ "--collector.ntp.protocol-version=4" "--no-collector.mdadm" ];
  };
}

_{blackbox-exporter}

The blackbox exporter allows blackbox probing of endpoints over HTTP, HTTPS, DNS, TCP, ICMP and gRPC.

Below is an example of blackbox-exporter monitoring a https-webservice:

≡︎ /etc/nixos/configuration.nix

{
  services.prometheus = {
      scrapeConfigs = [
        {
          job_name = "blackbox";
          metrics_path = "/probe";
          params.module = [ "http-2xx" ];
          static_configs = [
            {
              targets = [
                "https://example.com"
              ];
            }
          ];
          # The blackbox exporter needs to be passed the target as a parameter, this can be done with relabelling. (https://github.com/prometheus/blackbox_exporter#prometheus-configuration)
          relabel_configs = [
            {
              source_labels = [ "__address__" ];
              target_label = "__param_target";
            }
            {
              source_labels = [ "__param_target" ];
              target_label = "instance";
            }
            {
              target_label = "__address__";
              replacement = "localhost:9115";
            }
          ];
        }
    ];
    exporters = {
      blackbox = {
        enable = true;
        listenAddress = "[::]";
        configFile = builtins.toFile "blackbox-config" ''
          modules:
            http-2xx:
              prober: http
              timeout: 10s
              http:
                method: GET
                fail_if_not_ssl: true
                valid_status_codes: []
                headers:
                  User-Agent: "blackbox-exporter (contact: info@example.com)"
        '';
      };
    };
  };
}

Usage

The Prometheus service daemon can be enabled and configured by further options.

❄︎ /etc/nixos/configuration.nix

{
  services.prometheus.enable = true;
}

Another example:

❄︎ /etc/nixos/configuration.nix

{ config, pkgs, ... }:
{
  # https://wiki.nixos.org/wiki/Prometheus
  # https://nixos.org/manual/nixos/stable/#module-services-prometheus-exporters-configuration
  # https://github.com/NixOS/nixpkgs/blob/nixos-24.05/nixos/modules/services/monitoring/prometheus/default.nix
  services.prometheus = {
    enable = true;
    globalConfig.scrape_interval = "10s"; # "1m"
    scrapeConfigs = [
      {
        job_name = "node";
        static_configs = [{
          targets = [ "localhost:${toString config.services.prometheus.exporters.node.port}" ];
        }];
      }
    ];
  };
}

See also

• Grafana, the dashboarding tool often used with Prometheus.
• Grafana Loki, the equivalent of Prometheus for logs.