NixOS VM tests: Difference between revisions

(10 intermediate revisions by 9 users not shown)

Line 1:

WARNING: this page is mostly outdated as https://nixos.org/manual/nixos/stable/#chap-developing-the-test-driver mentions that <code>pkgs.nixosTest</code>, <code>testing-python.nix</code> and <code>make-test-python.nix</code> are outdated. Beside the documentation, you may also like https://nix.dev/tutorials/nixos/integration-testing-using-virtual-machines.html

The primary documentation for the [https://nixos.org/manual/nixos/stable/index.html#sec-nixos-tests NixOS VM testing framework] is in [https://nixos.org/manual/nixos/stable/index.html#sec-nixos-tests the NixOS manual], and in [https://nixos.org/manual/nixpkgs/unstable/#tester-runNixOSTest the Nixpkgs manual]. A tutorial can be found at [https://nix.dev/tutorials/nixos/integration-testing-using-virtual-machines].

The test infrastructure entry point is nixos/lib/testing.nix. Alternatively, for out-of-tree tests you can invoke it via Nixpkgs as the nixosTest function, which reuses your already evaluated Nixpkgs to generate your node configurations.

The test infra relies on the qemu build-vm code to generate virtual machines.

It will generate a test driver (a wrapper of nixos/lib/test-driver/test-driver.py) in charge of creating the network.

Line 11:

Line 16:

The wrapper `bin/nixos-run-vms` is in charge to start the driver with the correct VM script as arguments.

Once the driver is loaded, depending on the environment variables `tests` it will run in an interactive mode or run some ~~perl~~ code (`~~testScript~~`).

Once the driver is loaded, depending on the environment variables `tests` it will run in an interactive mode or run some python code (`test_script()`).

In interactive mode, you can run `start_all~~` followed by `join_all~~` to start and keep the VM alive

In interactive mode, you can run `start_all()` to start and keep the VM alive

== Connecting to an interactive VM via SSH ==

Add this to your test config:

interactive.sshBackdoor.enable = true;

Now you can connect to this VM via (on linux):

ssh -o User=root vsock/3

If you are on MacOS or have multiple VMs look at the shell output that do run the interactive tests:

[[File:Screenshot of interactive test just started.png|none|thumb|444x444px|Screenshot of just started nixos test for meilisearch (interactive)]]

== How to debug tests ? ==

Line 25:

Line 40:

== I don't see any prompt ? (qemu window pitch black) ==

Check the output for `malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "\x{0}\x{0}\x{0}\x{0}...") at /nix/store/1hkp2n6hz3ybf2rvkjkwrzgbjkrrakzl-update-users-groups.pl line 11`.

Check the output for

<code>malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "\x{0}\x{0}\x{0}\x{0}...") at /nix/store/1hkp2n6hz3ybf2rvkjkwrzgbjkrrakzl-update-users-groups.pl line 11`</code>

You should purge the state present in rm -rf /tmp/vm-state-<VM_NAME>

Line 219:

Line 237:

'';

}

== Tests that need multiple virtual machines ==

Tests can involve multiple virtual machines.

This example uses the use-case of a [https://en.m.wikipedia.org/wiki/REST REST] interface to a [https://www.postgresql.org/ PostgreSQL] database.

The following example Nix expression is adapted from [https://www.haskellforall.com/2020/11/how-to-use-nixos-for-lightweight.html How to use NixOS for lightweight integration tests].

This tutorial follows [https://postgrest.org/en/stable/tutorials/tut0.html PostgREST tutorial], a generic [https://restfulapi.net/ RESTful API] for PostgreSQL.

If you skim over the official tutorial, you'll notice there's quite a bit of setup in order to test if all the steps work.

The setup includes:

- A virtual machine named `server` running PostgreSQL and PostgREST.

- A virtual machine named `client` running HTTP client queries using `curl`.

- A `testScript` orchestrating testing logic between `client` and `server`.

The complete `postgrest.nix` file looks like the following:

let

# Pin Nixpkgs, as some packages are broken in the 22.11 release

nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/archive/0f8f64b54ed07966b83db2f20c888d5e035012ef.tar.gz";

pkgs = import nixpkgs { config = {}; overlays = []; };

# Single source of truth for all tutorial constants

database = "postgres";

schema = "api";

table = "todos";

username = "authenticator";

password = "mysecretpassword";

webRole = "web_anon";

postgrestPort = 3000;

# NixOS module shared between server and client

sharedModule = {

# Since it's common for CI not to have $DISPLAY available, explicitly disable graphics support

virtualisation.graphics = false;

};

in

pkgs.nixosTest {

# NixOS tests are run inside a virtual machine, and here you specify its system type

system = "x86_64-linux";

name = "postgres-test";

nodes = {

server = { config, pkgs, ... }: {

imports = [ sharedModule ];

networking.firewall.allowedTCPPorts = [ postgrestPort ];

services.postgresql = {

enable = true;

initialScript = pkgs.writeText "initialScript.sql" ''

create schema ${schema};

create table ${schema}.${table} (

id serial primary key,

done boolean not null default false,

task text not null,

due timestamptz

);

insert into ${schema}.${table} (task) values ('finish tutorial 0'), ('pat self on back');

create role ${webRole} nologin;

grant usage on schema ${schema} to ${webRole};

grant select on ${schema}.${table} to ${webRole};

create role ${username} inherit login password '${password}';

grant ${webRole} to ${username};

'';

};

users = {

mutableUsers = false;

users = {

# For ease of debugging the VM as the `root` user

root.password = "";

# Create a system user that matches the database user so that you

# can use peer authentication. The tutorial defines a password,

# but it's not necessary.

"${username}".isSystemUser = true;

};

systemd.services.postgrest = {

wantedBy = [ "multi-user.target" ];

after = [ "postgresql.service" ];

script =

let

configuration = pkgs.writeText "tutorial.conf" ''

db-uri = "postgres://${username}:${password}@localhost:${toString config.services.postgresql.port}/${database}"

db-schema = "${schema}"

db-anon-role = "${username}"

'';

in "${pkgs.haskellPackages.postgrest}/bin/postgrest ${configuration}";

serviceConfig.User = username;

};

client = {

imports = [ sharedModule ];

};

# Disable linting for simpler debugging of the testScript

skipLint = true;

testScript = ''

import json

import sys

start_all()

server.wait_for_open_port(${toString postgrestPort})

expected = [

{"id": 1, "done": False, "task": "finish tutorial 0", "due": None},

{"id": 2, "done": False, "task": "pat self on back", "due": None},

]

actual = json.loads(

client.succeed(

"${pkgs.curl}/bin/curl http://server:${toString postgrestPort}/${table}"

)

assert expected == actual, "table query returns expected content"

'';

}

Unlike the previous example, the virtual machines need an expressive name to distinguish them.

For this example we choose `client` and `server`.

Set up all machines and run the test script:

nix-build postgrest.nix

...

test script finished in 10.96s

cleaning up

killing client (pid 10)

killing server (pid 22)

(0.00 seconds)

/nix/store/bx7z3imvxxpwkkza10vb23czhw7873w2-vm-test-run-unnamed

== Further reading ==

* [https://nixcademy.com/posts/running-nixos-integration-tests-on-macos/ Run NixOS Integration Tests on macOS, Nixcademy]

* [https://nixcademy.com/posts/nixos-integration-tests/ Unveiling the Power of the NixOS Integration Test Driver (Part 1), Nixcademy]

* [https://nixcademy.com/posts/nixos-integration-tests-part-2/ Unveiling the Power of the NixOS Integration Test Driver (Part 2), Nixcademy]

[[Category:NixOS]]

@@ Line 1: / Line 1: @@
+WARNING: this page is mostly outdated as https://nixos.org/manual/nixos/stable/#chap-developing-the-test-driver mentions that  <code>pkgs.nixosTest</code>, <code>testing-python.nix</code> and <code>make-test-python.nix</code> are outdated. Beside the documentation, you may also like https://nix.dev/tutorials/nixos/integration-testing-using-virtual-machines.html
 The primary documentation for the [https://nixos.org/manual/nixos/stable/index.html#sec-nixos-tests NixOS VM testing framework] is in [https://nixos.org/manual/nixos/stable/index.html#sec-nixos-tests the NixOS manual], and in [https://nixos.org/manual/nixpkgs/unstable/#tester-runNixOSTest the Nixpkgs manual]. A tutorial can be found at [https://nix.dev/tutorials/nixos/integration-testing-using-virtual-machines].
 The test infrastructure entry point is nixos/lib/testing.nix. Alternatively, for out-of-tree tests you can invoke it via Nixpkgs as the nixosTest function, which reuses your already evaluated Nixpkgs to generate your node configurations.
 The test infra relies on the qemu build-vm code to generate virtual machines.
 It will generate a test driver (a wrapper of nixos/lib/test-driver/test-driver.py) in charge of creating the network.
@@ Line 11: / Line 16: @@
 The wrapper `bin/nixos-run-vms` is in charge to start the driver with the correct VM script as arguments.
-Once the driver is loaded, depending on the environment variables `tests` it will run in an interactive mode or run some perl code (`testScript`).
+Once the driver is loaded, depending on the environment variables `tests` it will run in an interactive mode or run some python code (`test_script()`).
-In interactive mode, you can run `start_all` followed by `join_all` to start and keep the VM alive
+In interactive mode, you can run `start_all()` to start and keep the VM alive
+== Connecting to an interactive VM via SSH ==
+Add this to your test config:
+ interactive.sshBackdoor.enable = true;
+Now you can connect to this VM via (on linux):
+ ssh -o User=root vsock/3
+If you are on MacOS or have multiple VMs look at the shell output that do run the interactive tests:
+[[File:Screenshot of interactive test just started.png|none|thumb|444x444px|Screenshot of just started nixos test for meilisearch (interactive)]]
 == How to debug tests ? ==
@@ Line 25: / Line 40: @@
 == I don't see any prompt ? (qemu window pitch black) ==
-Check the output for `malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "\x{0}\x{0}\x{0}\x{0}...") at /nix/store/1hkp2n6hz3ybf2rvkjkwrzgbjkrrakzl-update-users-groups.pl line 11`.
+Check the output for
+<code>malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "\x{0}\x{0}\x{0}\x{0}...") at /nix/store/1hkp2n6hz3ybf2rvkjkwrzgbjkrrakzl-update-users-groups.pl line 11`</code>
 You should purge the state present in rm -rf /tmp/vm-state-<VM_NAME>
@@ Line 219: / Line 237: @@
        '';
      }
+== Tests that need multiple virtual machines ==
+Tests can involve multiple virtual machines.
+This example uses the use-case of a [https://en.m.wikipedia.org/wiki/REST REST] interface to a [https://www.postgresql.org/ PostgreSQL] database.
+The following example Nix expression is adapted from [https://www.haskellforall.com/2020/11/how-to-use-nixos-for-lightweight.html How to use NixOS for lightweight integration tests].
+This tutorial follows [https://postgrest.org/en/stable/tutorials/tut0.html PostgREST tutorial], a generic [https://restfulapi.net/ RESTful API] for PostgreSQL.
+If you skim over the official tutorial, you'll notice there's quite a bit of setup in order to test if all the steps work.
+The setup includes:
+- A virtual machine named `server` running PostgreSQL and PostgREST.
+- A virtual machine named `client` running HTTP client queries using `curl`.
+- A `testScript` orchestrating testing logic between `client` and `server`.
+The complete `postgrest.nix` file looks like the following:
+  let
+    # Pin Nixpkgs, as some packages are broken in the 22.11 release
+    nixpkgs = fetchTarball "https://github.com/NixOS/nixpkgs/archive/0f8f64b54ed07966b83db2f20c888d5e035012ef.tar.gz";
+    pkgs = import nixpkgs { config = {}; overlays = []; };
+    # Single source of truth for all tutorial constants
+    database = "postgres";
+    schema        = "api";
+    table         = "todos";
+    username      = "authenticator";
+    password      = "mysecretpassword";
+    webRole       = "web_anon";
+    postgrestPort = 3000;
+    # NixOS module shared between server and client
+    sharedModule = {
+      # Since it's common for CI not to have $DISPLAY available, explicitly disable graphics support
+      virtualisation.graphics = false;
+    };
+  in
+    pkgs.nixosTest {
+      # NixOS tests are run inside a virtual machine, and here you specify its system type
+      system = "x86_64-linux";
+      name = "postgres-test";
+      nodes = {
+        server = { config, pkgs, ... }: {
+          imports = [ sharedModule ];
+          networking.firewall.allowedTCPPorts = [ postgrestPort ];
+          services.postgresql = {
+            enable = true;
+            initialScript = pkgs.writeText "initialScript.sql" ''
+              create schema ${schema};
+              create table ${schema}.${table} (
+                  id serial primary key,
+                  done boolean not null default false,
+                  task text not null,
+                  due timestamptz
+              );
+              insert into ${schema}.${table} (task) values ('finish tutorial 0'), ('pat self on back');
+              create role ${webRole} nologin;
+              grant usage on schema ${schema} to ${webRole};
+              grant select on ${schema}.${table} to ${webRole};
+              create role ${username} inherit login password '${password}';
+              grant ${webRole} to ${username};
+            '';
+          };
+          users = {
+            mutableUsers = false;
+            users = {
+              # For ease of debugging the VM as the `root` user
+              root.password = "";
+              # Create a system user that matches the database user so that you
+              # can use peer authentication. The tutorial defines a password,
+              # but it's not necessary.
+              "${username}".isSystemUser = true;
+            };
+          };
+          systemd.services.postgrest = {
+            wantedBy = [ "multi-user.target" ];
+            after = [ "postgresql.service" ];
+            script =
+              let
+                configuration = pkgs.writeText "tutorial.conf" ''
+                    db-uri = "postgres://${username}:${password}@localhost:${toString config.services.postgresql.port}/${database}"
+                    db-schema = "${schema}"
+                    db-anon-role = "${username}"
+                '';
+              in "${pkgs.haskellPackages.postgrest}/bin/postgrest ${configuration}";
+            serviceConfig.User = username;
+          };
+        };
+        client = {
+          imports = [ sharedModule ];
+        };
+      };
+      # Disable linting for simpler debugging of the testScript
+      skipLint = true;
+      testScript = ''
+        import json
+        import sys
+        start_all()
+        server.wait_for_open_port(${toString postgrestPort})
+        expected = [
+            {"id": 1, "done": False, "task": "finish tutorial 0", "due": None},
+            {"id": 2, "done": False, "task": "pat self on back", "due": None},
+        ]
+        actual = json.loads(
+            client.succeed(
+                "${pkgs.curl}/bin/curl http://server:${toString postgrestPort}/${table}"
+            )
+        )
+        assert expected == actual, "table query returns expected content"
+      '';
+  }
+Unlike the previous example, the virtual machines need an expressive name to distinguish them.
+For this example we choose `client` and `server`.
+Set up all machines and run the test script:
+  nix-build postgrest.nix
+  ...
+  test script finished in 10.96s
+  cleaning up
+  killing client (pid 10)
+  killing server (pid 22)
+  (0.00 seconds)
+  /nix/store/bx7z3imvxxpwkkza10vb23czhw7873w2-vm-test-run-unnamed
+== Further reading ==
+* [https://nixcademy.com/posts/running-nixos-integration-tests-on-macos/ Run NixOS Integration Tests on macOS, Nixcademy]
+* [https://nixcademy.com/posts/nixos-integration-tests/ Unveiling the Power of the NixOS Integration Test Driver (Part 1), Nixcademy]
+* [https://nixcademy.com/posts/nixos-integration-tests-part-2/ Unveiling the Power of the NixOS Integration Test Driver (Part 2), Nixcademy]
+[[Category:NixOS]]