Netdata

Netdata is a metrics tool, which comes with a lot of sane pre-configuration. It contains various plugins, which may need specific steps to be enabled.

Installation

Add the following to your NixOS configuration to setup and use Netdata:

❄︎ configuration.nix

{
  services.netdata = {
    enable = true;
    config.global = {
      "memory mode" = "ram";
      "debug log" = "none";
      "access log" = "none";
      "error log" = "syslog";
    };
  };
  networking.firewall.allowedTCPPorts = [ 19999 ];
}

🟆︎

Tip: In order to affect your NixOS system by your nix-language-specific changes you must first evaluate it:

$ nixos-rebuild switch --sudo

Netdata's basic instance will then be available at http://localhost:19999 on the local network.

Configuration

You may wish to aggregate multiple machines' Netdata information, in which case, you can subscribe to the Netdata Cloud service, or you can self-host Prometheus and Grafana as a self-hosted solution.

Adding node to cloud

Enable the Netdata service as described above.
override package to be built `withCloud`
When adding a new node in the web interface, you get a token; copy that token to /var/lib/netdata/cloud.d/token.
As root, run the netdata-claim.sh script.

# nix-shell -p netdata --run "netdata-claim.sh"

Declare claim token (option docs)

services.netdata = {
  package = pkgs.netdata.override { withCloud = true; };
  claimTokenFile = config.sops.secrets.netdata-token.path; # mounted by sops-nix, in this example
};

Streaming node setup

🛡︎︎

Security information: Ensure you choose the appropriate access control for your nodes.

Receiver node

❄︎ configuration.nix

{
  services.netdata.configDir."stream.conf" =
    let
      mkChildNode = apiKey: allowFrom: ''
        [${apiKey}]
          enabled = yes
          default history = &lt;A value of your choice&gt;
          default memory mode = dbengine
          health enabled by default = auto
          allow from = ${allowFrom}
      '';
    in pkgs.writeText "stream.conf" ''
      [stream]
      # This won't stream by itself, except if the receiver is a sender too, which is possible in the netdata model.
      enabled = no
      enable compression = yes

      # An allowed sender node
      ${mkChildNode "&lt;API key goes here&gt;" "&lt;Allowed IP goes here&gt;"}
    '';
}

Sender node

❄︎ configuration.nix

{
  services.netdata.configDir."stream.conf" = pkgs.writeText "stream.conf" ''
    [stream]
    enabled = yes
    destination = &lt;Receiver hostname or IP address goes here&gt;:19999
    api key = &lt;API key goes here&gt;
  '';
}

If you don't need any web UI and want to consume minimal resources on the sender node, use:

❄︎ configuration.nix

{
  services.netdata.config = {
    global = { "memory mode" = "none"; };
    web = {
      mode = "none";
      "accept a streaming request every seconds" = 0;
    };
  };
}

This way, it will neither spawn a web UI, nor store any metric locally.

Tips and Tricks

Modern Web UI

🟆︎

Tip: This package is unfree, and will require extra steps to install. You can read more about allowing unfree software in the Nixpkgs Manual.

Netdata comes with an old, unmaintained but open source web UI that is accessible at port 19999. Netdata Inc. will not fix any bugs in the old UI and it may to become more and more broken as time goes on. There is however, a newer, maintained but proprietary web UI that can be optionally enabled to replace the old UI. To use this new UI, override Netdata's package:

❄︎ configuration.nix

{
  nixpkgs.config.allowUnfreePredicate = pkg: builtins.elem (lib.getName pkg) [
    "netdata"
  ];
  services.netdata.package = pkgs.netdata.override {
    withCloudUi = true;
  };
}

Python Plugins

nvidia-smi

🟆︎

Tip: This package is unfree, and will require extra steps to install. You can read more about allowing unfree software in the Nixpkgs Manual.

To enable the nvidia-smi plugin, you have to ensure that nvidia-smi can be called by netdata:

❄︎ configuration.nix

{
  systemd.services.netdata.path = [ pkgs.linuxPackages.nvidia_x11 ];
  services.netdata.configDir."python.d.conf" = pkgs.writeText "python.d.conf" ''
    nvidia_smi: yes
  '';
}

samba

To enable the samba plugin, additional permissions and configurations will need to be set:

❄︎ configuration.nix

{
  services = {
    netdata.configDir."python.d.conf" = pkgs.writeText "python.d.conf" ''
      samba: yes
    '';
    samba.extraConfig = ''
      smbd profiling level = on
    '';
  };

  systemd.services.netdata = {
    path = [ pkgs.samba "/run/wrappers" ];
    serviceConfig.CapabilityBoundingSet = [ "CAP_SETGID" ];
  };

  security.sudo.extraConfig = ''
    netdata ALL=(root) NOPASSWD: ${pkgs.samba}/bin/smbstatus
  '';
}