Official NixOS Wiki

Talk:Proxmox Linux Container: Difference between revisions

← Older edit
VisualWikitext
imported>Vater
Vater (talk | contribs)
50 edits
 
(17 intermediate revisions by one other user not shown)
Line 33: Line 33:
| download the NixOS template
| download the NixOS template
|  
|  
* (if you want to switch to the ''Storage View'' and) go to one of your nodes where you can (find and) store ''CT Templates''.
* (if you want to switch to the <tt>Storage View</tt> and) go to one of your nodes where you can (find and) store <tt>CT Templates</tt>
* choose the ''Download from URL'' button
* choose the <tt>Download from URL</tt> button
** add the link for the NiixOS template file ''https://hydra.nixos.org/job/nixos/release-21.05/nixos.containerTarball.x86_64-linux/latest/download-by-type/file/system-tarball'' (to the text field of <tt>URL:</tt>)
** add the link for the NiixOS template file ''https://hydra.nixos.org/job/nixos/release-21.05/nixos.containerTarball.x86_64-linux/latest/download-by-type/file/system-tarball'' (to the text field of <tt>URL:</tt>)
** add a nice file name for the current downloadable build of the template (to the text field of <tt>File name:</tt>)
** add a nice file name for the current downloadable build of the template (to the text field of <tt>File name:</tt>)
Line 50: Line 50:
| create a (first) new NixOS container with Proxmox Virtual Environment  
| create a (first) new NixOS container with Proxmox Virtual Environment  
|  
|  
* (if you want to switch to the ''Folder View'' and go to ''Nodes'' and choose the your nodes where you create the container. (this note will be preselected as ''Node'' in the form.))
* (if you want to switch to the <tt>Folder View</tt> and go to <tt>Nodes</tt> and choose the your nodes where you create the container. (this note will be preselected as ''Node'' in the form.))
* choose the ''Create CT'' button
* choose the <tt>Create CT</tt> button
*: tab <tt>General</tt>
*: tab <tt>General</tt>
*:* (for the following we expect) ''1000'' is prefilled (to the text field of <tt>CT ID:</tt>) or you have added it (because it not already taken)
*:* (for the following we expect) ''1000'' is prefilled (to the text field of <tt>CT ID:</tt>) or you have added it (because it not already taken)
Line 62: Line 62:
*:** the name for the container (to the text field of <tt>Hostname:</tt>)
*:** the name for the container (to the text field of <tt>Hostname:</tt>)
*:** …
*:** …
*:* choose the ''Next'' button
*:* choose the <tt>Next</tt> button
*: tab <tt>Template</tt>
*: tab <tt>Template</tt>
*:* (for the following we expect) the entry ''cephfs'' is prefilled (at the drop down menu of <tt>Storage:</tt>)
*:* (for the following we expect) the entry ''cephfs'' is prefilled (at the drop down menu of <tt>Storage:</tt>)
*:* (for the following we expect you) find and choose the entry ''nixos-21.05_2021-10-10.tar.xz'' (at the drop down menu of <tt>Storage:</tt>)
*:* (for the following we expect you) find and choose the entry ''nixos-21.05_2021-10-10.tar.xz'' (at the drop down menu of <tt>Storage:</tt>)
*:* choose the ''Next'' button
*:* choose the <tt>Next</tt> button
*: tab <tt>Root Disk</tt>
*: tab <tt>Root Disk</tt>
*:* (for the following we expect) the entry ''storage'' is prefilled (at the drop down menu of <tt>Storage:</tt>)
*:* (for the following we expect) the entry ''storage'' is prefilled (at the drop down menu of <tt>Storage:</tt>)
*:* (for the following we expect) ''8'' is prefilled (to the text field of <tt>Disk size (GiB):</tt>)
*:* (for the following we expect) ''8'' is prefilled (to the text field of <tt>Disk size (GiB):</tt>)
*:* choose the ''Next'' button
*:* choose the <tt>Next</tt> button
*: tab <tt>CPU</tt>
*: tab <tt>CPU</tt>
*:* (for the following we expect) ''1'' is prefilled (to the text field of <tt>Cores:</tt>)
*:* (for the following we expect) ''1'' is prefilled (to the text field of <tt>Cores:</tt>)
*:* choose the ''Next'' button
*:* choose the <tt>Next</tt> button
*: tab <tt>Memory</tt>
*: tab <tt>Memory</tt>
*:* (for the following we expect) ''512'' is prefilled (to the text field of <tt>Memory (MiB):</tt>)
*:* (for the following we expect) ''512'' is prefilled (to the text field of <tt>Memory (MiB):</tt>)
*:* (for the following we expect) ''512'' is prefilled (to the text field of <tt>Swap (MiB):</tt>)
*:* (for the following we expect) ''512'' is prefilled (to the text field of <tt>Swap (MiB):</tt>)
*:* choose the ''Next'' button
*:* choose the <tt>Next</tt> button
*: tab <tt>Network</tt>
*: tab <tt>Network</tt>
*:* (for the following we expect) ''eth0'' is prefilled (to the text field of <tt>Name:</tt>)
*:* (for the following we expect) ''eth0'' is prefilled (to the text field of <tt>Name:</tt>)
Line 91: Line 91:
*:* (for the following we expect) the entry <tt>Static</tt> is prefilled (at the ratio button menu of <tt>IPv6:</tt>) and you have no network access for IPv6 avilibale
*:* (for the following we expect) the entry <tt>Static</tt> is prefilled (at the ratio button menu of <tt>IPv6:</tt>) and you have no network access for IPv6 avilibale
*:*: ?!? if you have network for IPv6 and you want to have access to your network for IPv6 you have fill out the text field of <tt>IPv6/CIDR:</tt> and the text field of <tt>Gateway (IPv6):</tt>) ?!?
*:*: ?!? if you have network for IPv6 and you want to have access to your network for IPv6 you have fill out the text field of <tt>IPv6/CIDR:</tt> and the text field of <tt>Gateway (IPv6):</tt>) ?!?
*:* choose the ''Next'' button
*:* choose the <tt>Next</tt> button
*: tab <tt>DNS</tt>
*: tab <tt>DNS</tt>
*:* ?!? emty ?!?
*:* <s>(for the following we expect) (the text field of <tt>DNS domain:</tt>) is emtpy (and so prefilled  with <tt>use host settings</tt>)</s>
*:* …
*:* <s>(for the following we expect) (the text field of <tt>DNS servers:</tt>) is emtpy (and so prefilled  with <tt>use host settings</tt>)</s>
*:* choose the ''Next'' button
*:* choose the <tt>Next</tt> button
*: tab <tt>Confirm</tt>
*: tab <tt>Confirm</tt>
*:: (optional) check the configuration (keys with values)
*:: (optional) check the configuration (keys with values)
*:* choose the ''Finish'' button
*:* choose the <tt>Finish</tt> button
<pre>
<pre>
/dev/rbd0
/dev/rbd0
Line 131: Line 131:
|  
|  
: <code>pct create </code>…
: <code>pct create </code>…
: <s><code>pct create 1000 --arch amd64 --description nixos-template cephfs:vztmpl/nixos-21.05_2021-10-10.tar.xz --ostype unmanaged --net0 name=eth0 --storage storage --unprivileged 1</code></s>
: <s><code>pct create 1000 cephfs:vztmpl/nixos-21.05_2021-10-10.tar.xz --ostype unmanaged --net0 name=eth0,firewall=1,ip=dhcp,bridge=vmbr0 --storage storage --unprivileged 1 --features nesting=1</code></s>
<pre></pre>
<pre></pre>
----
----
Line 138: Line 138:
<pre>
<pre>
arch: amd64
arch: amd64
features: nesting=1
hostname: CT1000
hostname: CT1000
memory: 1024
memory: 512
net0: name=eth0
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=1E:D8:FE:E9:F1:71,ip=dhcp,type=veth
ostype: unmanaged
ostype: unmanaged
rootfs: storage:vm-1000-disk-0,size=4G
rootfs: storage:vm-1000-disk-0,size=4G
Line 151: Line 152:
|  
|  
----
----
<!--
: <code>ping -c 5 cache.nixos.org</code>
<pre>
PING dualstack.v2.shared.global.fastly.net (151.101.114.217) 56(84) bytes of data.
64 bytes from 151.101.114.217 (151.101.114.217): icmp_seq=1 ttl=57 time=14.3 ms
64 bytes from 151.101.114.217 (151.101.114.217): icmp_seq=2 ttl=57 time=14.3 ms
64 bytes from 151.101.114.217 (151.101.114.217): icmp_seq=3 ttl=57 time=14.3 ms
64 bytes from 151.101.114.217 (151.101.114.217): icmp_seq=4 ttl=57 time=14.3 ms
64 bytes from 151.101.114.217 (151.101.114.217): icmp_seq=5 ttl=57 time=14.3 ms
--- dualstack.v2.shared.global.fastly.net ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4003ms
rtt min/avg/max/mdev = 14.302/14.312/14.325/0.009 ms
</pre>
-->
|-
|-
| customize the (pve) lxc startup setup routine (on every node where you want to start a NixOS container)
| customize the (pve) lxc startup setup routine (on every node where you want to start a NixOS container)
Line 207: Line 223:
|  
|  
|  
|  
[[NixOS]] is using "Sandboxing" by default.<ref>https://search.nixos.org/options?query=nix.useSandbox</ref> Therefore the option <code>nesting</code> for the  container on Proxmox Virtual Environment for NixOS must be acitivated.
----
if nesting is not acitivated
(try to) update (download and build) Nix expressions (<code>nix-env</code>) in a NixOS container on Proxmox Virtual Environment
: <code>nix-channel --update</code>
<pre>
unpacking channels...
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: while setting up the build environment: mounting /proc: Operation not permitted
error: program '/nix/store/rphxpqbsxgmykf8nyyr0pqi53nm78xa5-nix-2.3.15/bin/nix-env' failed with exit code 1
</pre>
(try to) rebuild NixOS
: <code>nixos-rebuild switch</code>
<pre>
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels/nixos' does not exist, ignoring
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: file 'nixpkgs/nixos' was not found in the Nix search path (add it using $NIX_PATH or -I), at (string):1:13
building Nix...
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels/nixos' does not exist, ignoring
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: file 'nixpkgs/nixos' was not found in the Nix search path (add it using $NIX_PATH or -I)
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels/nixos' does not exist, ignoring
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I)
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels/nixos' does not exist, ignoring
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: file 'nixpkgs/nixos/modules/installer/tools/nix-fallback-paths.nix' was not found in the Nix search path (add it using $NIX_PATH or -I)
/tmp/nixos-rebuild.qaFefR/nix
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels/nixos' does not exist, ignoring
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I)
building the system configuration...
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels/nixos' does not exist, ignoring
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
error: file 'nixpkgs/nixos' was not found in the Nix search path (add it using $NIX_PATH or -I)
</pre>
----
if nesting is acitivated
update (download and build) Nix expressions (<code>nix-env</code>) in a NixOS container on Proxmox Virtual Environment sucessfully
: <code>nix-channel --update</code>
<pre>
unpacking channels...
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
created 1 symlinks in user environment
</pre>
: <code>nix-channel --update</code>
<pre>
unpacking channels...
</pre>
rebuild NixOS sucessfully
: <code>nixos-rebuild switch</code>
<pre>
building Nix...
building the system configuration...
</pre>
<pre></pre>
<pre>
activating the configuration...
setting up /etc...
setting up tmpfiles
</pre>
----
----
how to configure that the option <code>nesting</code> is aciviated
(optional you can) check the (pve) lxc config file (and it should look like something like the following with ''nesting=1'' on the line ''features:'')
(optional you can) check the (pve) lxc config file (and it should look like something like the following with ''nesting=1'' on the line ''features:'')
<code>less /etc/pve/lxc/1000.conf</code>
<code>less /etc/pve/lxc/1000.conf</code>
<pre></pre>
<pre>
<pre>
arch: amd64
cores: 2
features: nesting=1
features: nesting=1
hostname: CT1000
memory: 1024
net0: name=eth0,bridge=vmbr0,hwaddr=A1:B2:C3:D4:E5:F6,ip=dhcp,type=veth
ostype: unmanaged
rootfs: storage:vm-1000-disk-0,size=4G
swap: 512
unprivileged: 1
</pre>
</pre>
<pre></pre>
After changing (adding) the option for <code>nesting</code> the container must be restarted.
|-
|-
| start the NixOS container with Proxmox Virtual Environment
| start the NixOS container with Proxmox Virtual Environment
|  
|  
* …
* choose the <tt>Start</tt> button
|  
|  
: <code>pct start </code>…
: <code>pct start 1000</code>
: <s><code>pct start 1000</code></s>
<pre>
<pre></pre>
WARN: old systemd (< v232) detected, container won't run in a pure cgroupv2 environment! Please see documentation -> container -> cgroup version.
Task finished with 1 warning(s)!
</pre>
: <code>pct status 1000</code>
<pre>
status: running
</pre>
|-
|-
| having a running NixOS container on Proxmox Virtual Environment
| having a running NixOS container on Proxmox Virtual Environment
Line 290: Line 381:
</pre>
</pre>
: <code>nixos-rebuild test</code>
: <code>nixos-rebuild test</code>
<!--
<pre>
building Nix...
building the system configuration...
</pre>
-->
<pre>
<pre>
building Nix...
building Nix...
Line 316: Line 413:
</pre>
</pre>
|-
|-
| fixing the configuration file for NixOS
| checking systemd
|  
|  
|  
|  
(optional) check the default configuration file for NixOS
: <code>systemctl list-units --failed</code>
: <code>cat /etc/nixos/configuration.nix</code>
<pre>
<pre>
{ config, pkgs, ... }:
  UNIT                  LOAD  ACTIVE SUB    DESCRIPTION           
● sys-kernel-debug.mount loaded failed failed Kernel Debug File System


{
LOAD  = Reflects whether the unit definition was properly loaded.
  imports = [ <nixpkgs/nixos/modules/virtualisation/lxc-container.nix> ];
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
1 loaded units listed.


 
}
</pre>
</pre>
|-
| fixing the configuration file for NixOS
|
|
: <code>nano /etc/nixos/configuration.nix</code>
: <code>nano /etc/nixos/configuration.nix</code>
<pre>
<pre>
Line 345: Line 446:
<pre>
<pre>
}
}
</pre>
: <code>nixos-rebuild switch</code>
<pre>
building Nix...
building the system configuration...
these derivations will be built:
  /nix/store/4c6mbv3y3i6m9qzv48j0ncy7163x3m7b-system-units.drv
  /nix/store/nfc7hz3shyrc1jm2nqsgh5m7izcj8psc-etc.drv
  /nix/store/fmx0v41zcyr7g38qjvdsdk5fifkjxfhx-nixos-system-nixos-21.05.3787.564cb4d81d4.drv
building '/nix/store/4c6mbv3y3i6m9qzv48j0ncy7163x3m7b-system-units.drv'...
building '/nix/store/nfc7hz3shyrc1jm2nqsgh5m7izcj8psc-etc.drv'...
building '/nix/store/fmx0v41zcyr7g38qjvdsdk5fifkjxfhx-nixos-system-nixos-21.05.3787.564cb4d81d4.drv'...
activating the configuration...
setting up /etc...
reloading user units for root...
setting up tmpfiles
</pre>
: <code>systemctl list-units --failed</code>
<pre>
  UNIT LOAD ACTIVE SUB DESCRIPTION
0 loaded units listed.
</pre>
</pre>
| https://github.com/NixOS/nixpkgs/issues/9735#issuecomment-783535726
| https://github.com/NixOS/nixpkgs/issues/9735#issuecomment-783535726
|-
|-
| …
| (optional) test changing the configuration of the NixOS container by installing a package (''ddate'') and disabling a (default running) service
|  
|  
|  
|  
: <code>nano /etc/nixos/configuration.nix</code>
<pre>
{
  imports = [ <nixpkgs/nixos/modules/virtualisation/lxc-container.nix> ];
</pre>
<pre></pre>
<pre>
services.openssh.enable = false;
</pre>
<pre></pre>
<pre>
  environment.systemPackages = with pkgs; [
    ddate
  ]; 
</pre>
<pre></pre>
<pre>
}
</pre>
: <code>nixos-rebuild switch</code>
<pre>
building Nix...
building the system configuration...
</pre>
<pre></pre>
: <code>ddate</code>
<pre>
Today is Prickle-Prickle, the 65th day of Bureaucracy in the YOLD 3187
</pre>
|-
|-
| …
| (optional) check status of systemd
|  
|  
|  
|  
: <code>systemctl status</code>
<pre>
● nixos
    State: running
    Jobs: 0 queued
  Failed: 0 units
    Since: Mon 2021-10-11 11:52:39 UTC; 1h 24min ago
  CGroup: /
          ├─user.slice
          │ └─user-0.slice
          │  ├─session-c1.scope
          │  │ ├─393 /nix/store/61z6l8p3f14hgz29j607bg1d37sn5d86-shadow-4.8.1/bin/login --
          │  │ ├─506 -bash
          │  │ └─527 top
          │  └─user@0.service
          │    └─init.scope
          │      ├─499 /run/current-system/systemd/lib/systemd/systemd --user --deserialize 17
          │      └─500 (sd-pam
          ├─.lxc
          │ ├─ 290 /bin/sh
          │ ├─3315 systemctl status
          │ └─3316 less
          ├─init.scope
          │ └─1 systemd
          └─system.slice
            ├─systemd-journald.service
            │ └─200 /nix/store/n5j5fjn60nhck658j9ab84k8n9z24n1r-systemd-247.6/lib/systemd/systemd-journald
            ├─nix-daemon.service
            │ └─473 nix-daemon --daemon
            ├─console-getty.service
            │ └─392 agetty --login-program /nix/store/61z6l8p3f14hgz29j607bg1d37sn5d86-shadow-4.8.1/bin/login --noclear --keep-baud console 115200,38400,9600 linux
            ├─dhcpcd.service
            │ └─468 dhcpcd: [master] [ip4] [ip6]
            ├─nscd.service
            │ └─450 nscd
            ├─system-container\x2dgetty.slice
            │ └─container-getty@1.service
            │  └─394 agetty --login-program /nix/store/61z6l8p3f14hgz29j607bg1d37sn5d86-shadow-4.8.1/bin/login --noclear --keep-baud pts/1 115200,38400,9600 vt220
            ├─dbus.service
            │ └─390 /nix/store/qxpxg30bmj99rvvsacqzkgayzvxz6bb1-dbus-1.12.20/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
            └─systemd-logind.service
              └─364 /nix/store/n5j5fjn60nhck658j9ab84k8n9z24n1r-systemd-247.6/lib/systemd/systemd-logind
</pre>
|-
|-
| …
| …
Line 366: Line 563:
; Is there a scenario where you would like to have a privileged (<code>unprivileged = 0</code>) container?: I do not know.
; Is there a scenario where you would like to have a privileged (<code>unprivileged = 0</code>) container?: I do not know.


==== Is there a cgroup2 only NixOS containers? ====
==== Is there a cgroup2 only NixOS container? ====


; Is there a configuration with cgroups versions 2 only - where cgroups version 1 is not needed - (so that is possible to rollback the manipulation of the pve lxc startup script (''/usr/share/perl5/PVE/LXC/Setup.pm''))? Or can we create such a (pre)build image (''tar'')?: I do not know.
; Is there a configuration with cgroups versions 2 only - where cgroups version 1 is not needed - (so that is possible to rollback the manipulation of the pve lxc startup script (<code>/usr/share/perl5/PVE/LXC/Setup.pm</code>))? Or can we create such a (pre)build image (''tar'')?: I do not know.


: <code>grep cgroup /proc/filesystems</code>
: <code>grep cgroup /proc/filesystems</code>
Line 389: Line 586:


https://git.proxmox.com/?p=pve-container.git;a=blob;f=src/PVE/LXC/Setup/Base.pm;h=a5b77d32f82747ea558d0398919414945b133dc0;hb=HEAD#l523
https://git.proxmox.com/?p=pve-container.git;a=blob;f=src/PVE/LXC/Setup/Base.pm;h=a5b77d32f82747ea558d0398919414945b133dc0;hb=HEAD#l523
: <code>nano /etc/nixos/configuration.nix</code>
<pre>
  systemd.enableUnifiedCgroupHierarchy = true;
</pre>
==== Why there is a ''Service'' section in the service manager configuration in a NixOS container? ====
<pre></pre>
<pre>
starting systemd...
/etc/systemd/system.conf:13: Unknown section 'Service'. Ignoring.
</pre>
<pre></pre>
<code>/etc/systemd/system.conf</code>
<pre></pre>
<pre>
[Service]
ProtectProc=default
ProtectControlGroups=no
ProtectKernelTunables=no
</pre>
<pre></pre>


== documentation ==
== documentation ==
Line 405: Line 626:
linux
linux
* …
* …
== articles ==
* https://blog.xirion.net/posts/nixos-proxmox-lxc/
*: with
** (PVE 6.3?)
** (NixOS 21.05?)
== upgrade (a PCT) to 21.11 ==
==== the folder ''/sbin/'' is missing ====
----
in your NixOS PCT
: <code>nix-channel --add https://nixos.org/channels/nixos-21.11 nixos</code>
: <code>nixos-rebuild switch --upgrade</code>
<pre></pre>
<pre>
ln: failed to create symbolic link '/sbin/init': No such file or directory
Activation script snippet 'installInitScript' failed (1)
</pre>
<pre></pre>
<pre>
warning: error(s) occurred while switching to the new configuration
</pre>
: <code>mkdir /sbin</code>
: <code>nixos-rebuild switch --upgrade</code>
<pre></pre>
----
on your PVE host
(for all the following starts of your NixOS PCT)
: <code>nano /etc/pve/lxc/1001.conf</code>
<pre></pre>
<pre>
#lxc.init.cmd: /init
lxc.init.cmd: /sbin/init
</pre>
<pre></pre>
----
== best (and easiest) way to get a "default" (working) shell in a pve ct after entering (<code>pct enter</code>) ==
:: <code>source /etc/set-environment</code>
: or
::: <code>/bin/sh -l</code>
:: or
::: <code>sh -l</code>
: or
:: <code>. /etc/profile</code>
or is possible to set an option in nixos or pve?
--[[User:Vater|Vater]] ([[User talk:Vater|talk]]) 17:47, 15 July 2024 (UTC)
Add topic
Retrieved from "https://wiki.nixos.org/wiki/Talk:Proxmox_Linux_Container"
Return to "Proxmox Linux Container" page.