Install NixOS on GCE: Difference between revisions

From NixOS Wiki
imported>Baracoder
Add a note on enable-oslogin
imported>Siriobalmelli
No edit summary
Line 1: Line 1:
NOTE: as of March 2020 this page is not accurate and many changes have taken place
This is a recipe for creating a NixOS machine on Google Compute Engine (GCE) which is part of [https://cloud.google.com/ Google Cloud Platform].
in NixOS and in GCE. There may be useful information here but following the instructions
will not lead to a happy running VM instance on GCE. Hopefully someone can figure out
how to fix it because there are updates in the source code. You may want to use [https://github.com/nix-community/nixos-generators nixos-generators] to create google cloud images.


This is a recipe for creating a NixOS machine on Google Compute Engine (GCE) which is part of [https://cloud.google.com/ Google Cloud Platform]. This tutorial assumes you have already set up and account and project under Google Cloud Platform.
This tutorial assumes you have already set up and account and project under Google Cloud Platform.


==  Preparing a NixOS image ==
This tutorial is confirmed from NixOS 19.03 onward only (see [https://github.com/NixOS/nixpkgs/blob/b1178783876f51864f03f68eb89e9e93656bc57b/nixos/doc/manual/release-notes/rl-1903.xml#L98 release notes for 19.03]).
To bootstrap the process, first start any Linux instance and log into it.
# From the Google Developers Console select Compute - Compute Engine - VM instances
# Press "Create an Instance" or "New instance" button
# Click on "Show advanced options"
# Select a ZONE close to you and any MACHINE TYPE (f1-micro is adequate)
# Select any linux image such as centos-7
# Make sure "Enable Compute Engine service account." is checked
# Under COMPUTE select "Read Write"
# Press "Create" button and wait until the VM is ready
# From the Google Developers Console select Compute - Compute Engine - VM instances press the "SSH" button on your new instance (usually named "instance-1") to log in


Next we will create a NixOS image. A source for NixOS images can be found the google storage bucket named [https://storage.cloud.google.com/nixos-images gs://nixos-images].
Note also that [https://github.com/NixOS/nixops nixops] provides automated tooling to deploy NixOS on GCE instances. This tutorial covers manually deploying a NixOS GCE instance.
<syntaxhighlight lang="console">
 
$ gsutil ls -l gs://nixos-images
== Obtain the <code>gsutil</code> utility ==
256556736  2014-12-17T10:51:00Z  gs://nixos-images/nixos-14.10pre-git-x86_64-linux.raw.tar.gz
290985235  2014-12-19T12:45:58Z  gs://nixos-images/nixos-14.12.542.4c9ef9f7-x86_64-linux.raw.tar.gz
</syntaxhighlight>


You can find more up-to-date images at [https://storage.cloud.google.com/nixos-cloud-images gs://nixos-cloud-images], but the bucket itself is not public- <u>the objects inside it are</u>.
<code>gsutil</code> is part of <code>google-cloud-sdk</code>; you can obtain this several ways:


This means that <code>gsutil ls gs://nixos-cloud-images</code> will yield an <code>AccessDeniedException</code>, but the following works:
# Using nixpkgs: <code>nix-env --install google-cloud-sdk</code>
<syntaxhighlight lang="console">
# Using an existing non-Nix GCE instance (eg Debian)
$ gsutil ls -l gs://nixos-cloud-images/nixos-image-18.09.1228.a4c4cbb613c-x86_64-linux.raw.tar.gz
# Following Google's instructions here: [https://cloud.google.com/sdk cloud.google.com]
344473043 2018-11-16T09:16:51Z  gs://nixos-cloud-images/nixos-image-18.09.1228.a4c4cbb613c-x86_64-linux.raw.tar.gz
TOTAL: 1 objects, 344473043 bytes (328.52 MiB)
</syntaxhighlight>


You can find a full list of available nixos-cloud-images at [https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/gce-images.nix <nixpkgs/nixos/modules/virtualisation/gce-images.nix>].
== Select a NixOS image ==


In this tutorial we will use gs://nixos-images/nixos-14.12.542.4c9ef9f7-x86_64-linux.raw.tar.gz and follow the [https://cloud.google.com/compute/docs/images?_ga=1.122328651.1179090775.1417532639#creating_an_image_from_a_tar_file documentation on how to create an image from a .tar.gz file].
Get a list of currently available GCE images with <code>gsutil ls -l gs://nixos-images</code>:


<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
$ gcloud compute images create nixos-14125424c9ef9f7-x86-64-linux --source-uri gs://nixos-images/nixos-14.12.542.4c9ef9f7-x86_64-linux.raw.tar.gz
$ gsutil ls -l gs://nixos-images
$ gcloud compute images describe nixos-14125424c9ef9f7-x86-64-linux
397747554  2020-05-03T02:36:23Z  gs://nixos-images/google-cloud-nixos-20.03.1639.73e73c7d6b5.raw.tar.gz
...
256556736  2014-12-17T10:51:00Z  gs://nixos-images/nixos-14.10pre-git-x86_64-linux.raw.tar.gz
status: READY
291081495  2015-01-16T16:36:46Z  gs://nixos-images/nixos-14.12.323.91643074-x86_64-linux.raw.tar.gz
290985235  2014-12-19T12:45:58Z  gs://nixos-images/nixos-14.12.542.4c9ef9f7-x86_64-linux.raw.tar.gz
TOTAL: 4 objects, 1236371020 bytes (1.15 GiB)
</syntaxhighlight>
</syntaxhighlight>


You can now log out of your VM and delete the instance
Other images are also listed at [https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/gce-images.nix <nixpkgs/nixos/modules/virtualisation/gce-images.nix>].
# $ logout
# From the Google Developers Console select Compute - Compute Engine - VM instances check your instance and then press the "Delete" button at the top of the page.


== Starting a NixOS image ==
== Import an image into your project ==
Now that the image is made we can start as many NixOS VMs as we like.


=== (Strongly Recommended) Generate SSH host keys ===
#In [https://console.cloud.google.com/compute/images console.cloud.google.com/compute/images], select <code>CREATE IMAGE</code>:
Before beginning we should generate unique SSH host keys for our new VM so that when will be able to authenticate our SSH connection. The example below generates keys <code>/dev/shm/ssh_host_ecdsa_key</code> and <code>/dev/shm/ssh_host_ecdsa_key.pub</code> files.
## <b>Name</b> : <i>nixos-20-03</i>
## <b>Source</b> : <i>Cloud Storage file</i>
## <b>Cloud Storage file</b> : <i>gs://nixos-images/google-cloud-nixos-20.03.1639.73e73c7d6b5.raw.tar.gz</i>
#Click <code>Create</code>


<syntaxhighlight lang="bash">
== Create a VM instance ==
$ ssh-keygen -N '' -C '' -t ecdsa -f /dev/shm/ssh_host_ecdsa_key
Generating public/private ecdsa key pair.
Your identification has been saved in /dev/shm/ssh_host_ecdsa_key.
Your public key has been saved in /dev/shm/ssh_host_ecdsa_key.pub.
The key fingerprint is:
92:2a:e9:28:1a:cd:43:71:31:36:f2:8e:6e:fa:13:c4
The key's randomart image is:
+--[ECDSA  256]---+
|  . =           |
|  + +          |
| .. o            |
|  E=   .        |
| .o . o S        |
| =.. . .        |
|. O..            |
|.*.o            |
|*.o.            |
+-----------------+
</syntaxhighlight>


'''Take note of your unique key fingerprint.'''
#In [https://console.cloud.google.com/compute/instances console.cloud.google.com/compute/instances], select <code>CREATE INSTANCE</code>
## <b>Boot disk</b> : <i>Custom images</i>
### <b>Image</b> : <i>nixos-20-03</i>
## Important: do not add SSH keys, NixOS is set up for [https://cloud.google.com/compute/docs/instances/managing-instance-access Google OS Login]
## <b>Metadata</b>
### <b>key</b> : <i>enable-oslogin</i>
### <b>value</b> : <i>TRUE</i>
# Click <b>Create</b>
# Wait until your VM instance is ready
# Under <i>Connect</i>, click <b>SSH</b>


=== Create your VM ===
== Optional: add user account ==
Now we are ready to create a NixOS VM.
# From the Google Developers Console select Compute - Compute Engine - VM instances
# Press "Create an Instance" or "New instance" button
# Click on "Show advanced options"
# (Strongly Recommended) Under METADATA, create a Key called ssh_host_ecdsa_key and copy the contents of your <code>/dev/shm/ssh_host_ecdsa_key</code> to the corresponding Value if you generated SSH host keys.
# (Strongly Recommended) Under METADATA, create a Key called ssh_host_ecdsa_key_pub and copy the contents of your <code>/dev/shm/ssh_host_ecdsa_key.pub</code> to the corresponding Value if you generated SSH host keys.
# Under SSH KEYS, copy the contents of your desktop computer's <code>~/.ssh/id_rsa.pub</code> file into the box that says "Enter entire key data"
# Under BOOT SOURCE, make sure "New disk form image" is selected
# Under IMAGE, select your nixos image, e.g. "nixos-14125424c9ef9f7-x86-64-linux".
# Set any additional options as you see fit.
# Press the "Create" button and wait until the VM is ready


=== Connect to your VM ===
Once you are logged into your NixOS machine, you can create a user account for yourself with administrator privileges:
Notwithstanding the username associated with your SSH, you must log in as root to the NixOS machine. Also, you will need to ssh in from your computer's terminal. Below replace <code>XXX.XXX.XXX.XXX</code> with the EXTERNAL IP address listed for your VM instance. (Strongly Recommended) Verify the ECDSA key fingerprint is the same as the one you generated.


<syntaxhighlight lang="bash">
1. chmod u+w /etc/nixos/configuration.nix
$ ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=ask root@XXX.XXX.XXX.XXX
The authenticity of host '130.211.149.218 (130.211.149.218)' can't be established.
ECDSA key fingerprint is 92:2a:e9:28:1a:cd:43:71:31:36:f2:8e:6e:fa:13:c4.
Are you sure you want to continue connecting (yes/no)? yes
Enter passphrase for key '.../.ssh/id_rsa':


[root@instance-1:~]#
2. nano -w /etc/nixos/configuration.nix
</syntaxhighlight>
 
Once you are logged into your NixOS machine, I recommend that you create an user account for yourself with administrator privileges.
 
# chmod u+w /etc/nixos/configuration.nix
# nano -w /etc/nixos/configuration.nix
#Add the follow to the configuration:


3. Add the following to the configuration:
<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
security.sudo.wheelNeedsPassword = false;
security.sudo.wheelNeedsPassword = false;
Line 123: Line 73:
</syntaxhighlight>
</syntaxhighlight>


After you save this file run <code>nixos-rebuild switch --upgrade</code>. Once that is complete reboot and log back in with your user account. (Strongly Recommended) Again verify the ECDSA key fingerprint is the same as the one you generated. If you plan to keep this instance running for a long time you may removed the <code>-o UserKnownHostsFile=/dev/null</code> option to SSH to add the host to your <code>~/.ssh/known_hosts</code> file, however be advised that IP addresses for GCE instances are often reused, so when you create and destroy instances you may end up with stale keys in your known_hosts file.
4. Save this file and run <code>nixos-rebuild switch --upgrade</code>


<syntaxhighlight lang="bash">
5. Reboot and log back in with your user account
$ ssh -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=ask <your-username>@XXX.XXX.XXX.XXX
The authenticity of host '130.211.149.218 (130.211.149.218)' can't be established.
ECDSA key fingerprint is 92:2a:e9:28:1a:cd:43:71:31:36:f2:8e:6e:fa:13:c4.
Are you sure you want to continue connecting (yes/no)? yes
Enter passphrase for key '.../.ssh/id_rsa':


[<your-username>@instance-1:~]$
== Snapshots ==
</syntaxhighlight>


At this point you may want to snapshot this image and use this snapshot to make future VMs. You should also delete the <code>/dev/shm/ssh_host_ecdsa_key</code> and <code>/dev/shm/ssh_host_ecdsa_key.pub</code> files from your home machine if you generated them.
At this point you may want to snapshot this image and use this snapshot to make future VMs.


== Bootstrapping a NixOS image from the build of your choice ==
== Bootstrapping a NixOS image from the build of your choice ==
You normally can use a preexisting NixOS image, such as the ones found in gs://nixos-images and gs://nixos-cloud-images. Sometimes the images there haven't been updated in a long time, or you need a newer one to work around a bug. If you need to make a new one, here's how. Perform these steps from any Linux machine.
 
You normally can use a preexisting NixOS image, such as the ones found in <code>gs://nixos-images</code> and <code>gs://nixos-cloud-images</code>. Sometimes the images there haven't been updated in a long time, or you need a newer one to work around a bug. If you need to make a new one, here's how.
 
Perform these steps from any Linux machine.


You'll need setup the credentials for Google Cloud (via <code>gsutil config</code> or similar- if you have a GCE service account and have created an associated access key, the following is nice and quick: <code>readlink -f ./relative/path/to/key | gsutil config -e</code>, though it will prompt you for permission to lock down the file permissions on the key if they currently allow others to read the key- Note: the service account must be configured with a Role that allows it to write to your storage bucket), with the Cloud Storage and Compute Engine APIs enabled. You'll also need a Storage bucket. These steps will assume you've already made a bucket and it's named <code>gs://example</code>.
You'll need setup the credentials for Google Cloud (via <code>gsutil config</code> or similar- if you have a GCE service account and have created an associated access key, the following is nice and quick: <code>readlink -f ./relative/path/to/key | gsutil config -e</code>, though it will prompt you for permission to lock down the file permissions on the key if they currently allow others to read the key- Note: the service account must be configured with a Role that allows it to write to your storage bucket), with the Cloud Storage and Compute Engine APIs enabled. You'll also need a Storage bucket. These steps will assume you've already made a bucket and it's named <code>gs://example</code>.

Revision as of 17:23, 6 November 2020

This is a recipe for creating a NixOS machine on Google Compute Engine (GCE) which is part of Google Cloud Platform.

This tutorial assumes you have already set up and account and project under Google Cloud Platform.

This tutorial is confirmed from NixOS 19.03 onward only (see release notes for 19.03).

Note also that nixops provides automated tooling to deploy NixOS on GCE instances. This tutorial covers manually deploying a NixOS GCE instance.

Obtain the gsutil utility

gsutil is part of google-cloud-sdk; you can obtain this several ways:

  1. Using nixpkgs: nix-env --install google-cloud-sdk
  2. Using an existing non-Nix GCE instance (eg Debian)
  3. Following Google's instructions here: cloud.google.com

Select a NixOS image

Get a list of currently available GCE images with gsutil ls -l gs://nixos-images:

$ gsutil ls -l gs://nixos-images
 397747554  2020-05-03T02:36:23Z  gs://nixos-images/google-cloud-nixos-20.03.1639.73e73c7d6b5.raw.tar.gz
 256556736  2014-12-17T10:51:00Z  gs://nixos-images/nixos-14.10pre-git-x86_64-linux.raw.tar.gz
 291081495  2015-01-16T16:36:46Z  gs://nixos-images/nixos-14.12.323.91643074-x86_64-linux.raw.tar.gz
 290985235  2014-12-19T12:45:58Z  gs://nixos-images/nixos-14.12.542.4c9ef9f7-x86_64-linux.raw.tar.gz
TOTAL: 4 objects, 1236371020 bytes (1.15 GiB)

Other images are also listed at <nixpkgs/nixos/modules/virtualisation/gce-images.nix>.

Import an image into your project

  1. In console.cloud.google.com/compute/images, select CREATE IMAGE:
    1. Name : nixos-20-03
    2. Source : Cloud Storage file
    3. Cloud Storage file : gs://nixos-images/google-cloud-nixos-20.03.1639.73e73c7d6b5.raw.tar.gz
  2. Click Create

Create a VM instance

  1. In console.cloud.google.com/compute/instances, select CREATE INSTANCE
    1. Boot disk : Custom images
      1. Image : nixos-20-03
    2. Important: do not add SSH keys, NixOS is set up for Google OS Login
    3. Metadata
      1. key : enable-oslogin
      2. value : TRUE
  2. Click Create
  3. Wait until your VM instance is ready
  4. Under Connect, click SSH

Optional: add user account

Once you are logged into your NixOS machine, you can create a user account for yourself with administrator privileges:

1. chmod u+w /etc/nixos/configuration.nix

2. nano -w /etc/nixos/configuration.nix

3. Add the following to the configuration:

security.sudo.wheelNeedsPassword = false;
users.extraUsers.<your-username> = {
  createHome = true;
  home = "/home/<your-username>";
  description = "<your-name>";
  group = "users"; 
  extraGroups = [ "wheel" ];
  useDefaultShell = true;
  openssh.authorizedKeys.keys = [ "<contents of your ~/.ssh/id_rsa.pub>" ];
};

4. Save this file and run nixos-rebuild switch --upgrade

5. Reboot and log back in with your user account

Snapshots

At this point you may want to snapshot this image and use this snapshot to make future VMs.

Bootstrapping a NixOS image from the build of your choice

You normally can use a preexisting NixOS image, such as the ones found in gs://nixos-images and gs://nixos-cloud-images. Sometimes the images there haven't been updated in a long time, or you need a newer one to work around a bug. If you need to make a new one, here's how.

Perform these steps from any Linux machine.

You'll need setup the credentials for Google Cloud (via gsutil config or similar- if you have a GCE service account and have created an associated access key, the following is nice and quick: readlink -f ./relative/path/to/key | gsutil config -e, though it will prompt you for permission to lock down the file permissions on the key if they currently allow others to read the key- Note: the service account must be configured with a Role that allows it to write to your storage bucket), with the Cloud Storage and Compute Engine APIs enabled. You'll also need a Storage bucket. These steps will assume you've already made a bucket and it's named gs://example. Prepare a local copy of the nixpkgs repository in the state you want to build from. If you want to build a released version, this means checking out one of the release branches from the nixpkgs-channels repository. Make sure you haven't left any unwanted local changes in it. These examples assume you've checked it out at /home/example/nixpkgs-clean.

$ BUCKET_NAME=example /home/example/nixpkgs-clean/nixos/maintainers/scripts/gce/create-gce.sh

This will create an image and upload it to the bucket example

Note: If you build an image from a commit later then this one, you will need to add enable-oslogin = "TRUE" to the instance metadata, to be able to login.