Nixpkgs/Create and debug packages: Difference between revisions

Line 4:

Writing packages is covered in https://nixos.org/manual/nixpkgs/stable/#chap-quick-start and writing modules in http://nixos.org/nixos/manual

If you've read the manual and still don't know how to go about creating a package read on.

The nix repositories are hosted here: https://github.com/nixos

Line 21:

Line 23:

This is pretty much the standard way to use github, so if you have trouble using git or github any general guide on these should get you going, or just ask on the NixOS IRC channel. The rest of this guide deals with the "Hack hack hack" step :)

== Rough process for creating a package ==

There are different steps here depending on whether you're building from source or packaging an existing binary. There are some common steps too.

=== Packages from source code ===

# Read the repo build instructions. Many times the instructions won't be 100% clear in the README, but there might be some script for CI that will be more precise. This should give you a rough idea of how hard it will be to package it. For example, if it uses a standard, vanilla build, it will probably be easy to package. If on the contrary there are no CI scripts, or no build instructions, or the build instructions are complicated, things could be rough.

# Look in nixpkgs for a similar package to use as reference. Many times you can just copy/paste a similar package, change the <code>src</code> attribute and things will just work. E.g. if you're packaging a project written in Go, find a package for an existing Go application. Each language has its own supporting Nix functions and a more or less standard way of dealing with things. For example Go has <code>buildGoPackage</code>. Rust has <code>buildRustPackage</code>. Python has <code>buildPythonApplication</code>. The official documentation for these is generally good. These functions will typically do most of the heavy lifting and you probably just have to point them to the repo and they will do the rest. There are also specific functions for wrapping e.g. GNOME applications.

It pays off to have a look at the [https://nixos.org/manual/nixpkgs/stable/#chap-language-support table of contents for these functions in the docs].

# If there isn't a specific function for the build/language you'll need to use stdenv.mkDerivation directly but don't despair (yet!). mkDerivation has built-in support for GNU make, CMake, and possibly others too (?) so you probably won't have to spell out every single command necessary to build the project.

# Figure out at least some dependencies from the project repo. Doesn't have to be perfect, but something to get you started. See if they're available in nixpkgs. If any dependency is missing you'll need to package that as well! Common libraries are usually available though.

# Prototype / iterate the build with nix-shell. Run <code>nix-shell -p dep1 dep2 dep3...</code> then run the build manually on the command line. Hopefully this will let you quickly identify missing dependencies and maybe some build quirks. Again this doesn't need to be perfect, don't spend too much time on this.

# Write the default.nix. <br/>

Do it in some empty local directory and at the top just say <code>with import <nixpkgs> {};</code>, don't worry too much about declaring every dependency as a parameter for now as it will only slow you down. Again, use some similar package as reference.

# Build it with <code>nix build -L</code>. Iterate on tweaking default.nix and running nix build until it builds. At this stage you will start dealing with build quirks of the project, if the are any. Compile-time errors will hopefully point out what you're missing. For example [https://github.com/NixOS/nixpkgs/blob/643ce4bd0f057bc0b90f0faebeb83a3b14f01674/pkgs/tools/package-management/micromamba/default.nix#L6-L10 micromamba needs a specialized build of libsolv].

# Read on below for further steps.

=== Packages from binaries ===

# There's probably a package for it for some other distro. Use that package definition to figure out the dependencies. For example, if you have a deb package you can view its dependencies by running <code>dpkg -I <package.deb></code>. [https://aur.archlinux.org/packages/ Arch packages] can also be useful to look up for reference (view the package's PKGBUILD):

# Sometimes the definitions for other distros won't be enough by nix's standards. If that's the case, use <code>ldd</code> and/or <code>strace</code> to find the rest of the dependencies. If you're not familiar with <code>ldd</code>/<code>strace</code> see https://unix.stackexchange.com/questions/120015/how-to-find-out-the-dynamic-libraries-executables-loads-when-run

# See how other nix binary packages deal with dependencies. For example [https://github.com/NixOS/nixpkgs/search?q=%22dpkg+-x%22 nix packages based on deb packages].

# If the application contains some helper executable or vendored dlopen'd library you will probably need to give the nix treatment to it as well. For example, [https://github.com/NixOS/nixpkgs/blob/42c154d332eb4eb17c74b587c1d4c2fcc3042ba1/pkgs/applications/editors/jetbrains/default.nix#L196-L200 JetBrains Rider vendors dotnet so it needs to be replaced with the dotnet nix package].

# Because there's no real build step here you'll have to rely more on testing the actual execution of the package.

=== Both source code packages and binary packages ===

# Once you have the package building successfully, it's time to start testing the output. Run <code>nix run -L</code>, this will give you a shell where the application executable should be in the PATH. If the executable is not in the PATH, you might need to override <code>installPhase</code>. Try to test as many functions of the application as you can. Most of the time only making sure that the application starts up will not be enough.

# Now that your package builds and runs, it's time to move it to nixpkgs. Read https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md , make sure your package is up to the standards e.g. add a suitable [https://nixos.org/manual/nixpkgs/stable/#sec-standard-meta-attributes <code>meta</code> section].

# Git clone https://github.com/NixOS/nixpkgs , figure out the best category / directory for the application (within https://github.com/NixOS/nixpkgs/tree/master/pkgs/ ), create the directory for your application, and move your default.nix there.

# If you used <code>with import <nixpkgs> {};</code> to iterate more quickly, now is the time to replace that with the actual dependencies as an attribute set at the beginning of the file e.g. <code>{ lib, stdenv, fetchFromGitHub }:</code>

# Add the package to the top level declaration of packages. Most of the time this will be https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix .

# If this is your first package in nixpkgs, add yourself in https://github.com/NixOS/nixpkgs/blob/master/maintainers/maintainer-list.nix

# Read on about the final steps of branching and sending your PR in https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md .

== How to install from the local repository ==

@@ Line 4: / Line 4: @@
 Writing packages is covered in https://nixos.org/manual/nixpkgs/stable/#chap-quick-start and writing modules in http://nixos.org/nixos/manual
+If you've read the manual and still don't know how to go about creating a package read on.
 The nix repositories are hosted here: https://github.com/nixos
@@ Line 21: / Line 23: @@
 This is pretty much the standard way to use github, so if you have trouble using git or github any general guide on these should get you going, or just ask on the NixOS IRC channel. The rest of this guide deals with the "Hack hack hack" step :)
+== Rough process for creating a package ==
+There are different steps here depending on whether you're building from source or packaging an existing binary.  There are some common steps too.
+=== Packages from source code ===
+# Read the repo build instructions. Many times the instructions won't be 100% clear in the README, but there might be some script for CI that will be more precise. This should give you a rough idea of how hard it will be to package it. For example, if it uses a standard, vanilla build, it will probably be easy to package. If on the contrary there are no CI scripts, or no build instructions, or the build instructions are complicated, things could be rough.
+# Look in nixpkgs for a similar package to use as reference. Many times you can just copy/paste a similar package, change the <code>src</code> attribute and things will just work. E.g. if you're packaging a project written in Go, find a package for an existing Go application. Each language has its own supporting Nix functions and a more or less standard way of dealing with things. For example Go has <code>buildGoPackage</code>. Rust has <code>buildRustPackage</code>. Python has <code>buildPythonApplication</code>. The official documentation for these is generally good. These functions will typically do most of the heavy lifting and you probably just have to point them to the repo and they will do the rest. There are also specific functions for wrapping e.g. GNOME applications.
+It pays off to have a look at the [https://nixos.org/manual/nixpkgs/stable/#chap-language-support table of contents for these functions in the docs].
+# If there isn't a specific function for the build/language you'll need to use stdenv.mkDerivation directly but don't despair (yet!). mkDerivation has built-in support for GNU make, CMake, and possibly others too (?) so you probably won't have to spell out every single command necessary to build the project.
+# Figure out at least some dependencies from the project repo. Doesn't have to be perfect, but something to get you started. See if they're available in nixpkgs. If any dependency is missing you'll need to package that as well! Common libraries are usually available though.
+# Prototype / iterate the build with nix-shell. Run <code>nix-shell -p dep1 dep2 dep3...</code> then run the build manually on the command line. Hopefully this will let you quickly identify missing dependencies and maybe some build quirks. Again this doesn't need to be perfect, don't spend too much time on this.
+# Write the default.nix. <br/>
+Do it in some empty local directory and at the top just say <code>with import <nixpkgs> {};</code>, don't worry too much about declaring every dependency as a parameter for now as it will only slow you down. Again, use some similar package as reference.
+# Build it with <code>nix build -L</code>. Iterate on tweaking default.nix and running nix build until it builds. At this stage you will start dealing with build quirks of the project, if the are any. Compile-time errors will hopefully point out what you're missing. For example [https://github.com/NixOS/nixpkgs/blob/643ce4bd0f057bc0b90f0faebeb83a3b14f01674/pkgs/tools/package-management/micromamba/default.nix#L6-L10 micromamba needs a specialized build of libsolv].
+# Read on below for further steps.
+=== Packages from binaries ===
+# There's probably a package for it for some other distro. Use that package definition to figure out the dependencies. For example, if you have a deb package you can view its dependencies by running <code>dpkg -I <package.deb></code>. [https://aur.archlinux.org/packages/ Arch packages] can also be useful to look up for reference (view the package's PKGBUILD):
+# Sometimes the definitions for other distros won't be enough by nix's standards. If that's the case, use <code>ldd</code> and/or <code>strace</code> to find the rest of the dependencies. If you're not familiar with <code>ldd</code>/<code>strace</code> see https://unix.stackexchange.com/questions/120015/how-to-find-out-the-dynamic-libraries-executables-loads-when-run
+# See how other nix binary packages deal with dependencies. For example [https://github.com/NixOS/nixpkgs/search?q=%22dpkg+-x%22 nix packages based on deb packages].
+# If the application contains some helper executable or vendored dlopen'd library you will probably need to give the nix treatment to it as well. For example, [https://github.com/NixOS/nixpkgs/blob/42c154d332eb4eb17c74b587c1d4c2fcc3042ba1/pkgs/applications/editors/jetbrains/default.nix#L196-L200 JetBrains Rider vendors dotnet so it needs to be replaced with the dotnet nix package].
+# Because there's no real build step here you'll have to rely more on testing the actual execution of the package.
+=== Both source code packages and binary packages ===
+# Once you have the package building successfully, it's time to start testing the output. Run <code>nix run -L</code>, this will give you a shell where the application executable should be in the PATH. If the executable is not in the PATH, you might need to override <code>installPhase</code>. Try to test as many functions of the application as you can. Most of the time only making sure that the application starts up will not be enough.
+# Now that your package builds and runs, it's time to move it to nixpkgs. Read https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md , make sure your package is up to the standards e.g. add a suitable [https://nixos.org/manual/nixpkgs/stable/#sec-standard-meta-attributes <code>meta</code> section].
+# Git clone https://github.com/NixOS/nixpkgs , figure out the best category / directory for the application (within https://github.com/NixOS/nixpkgs/tree/master/pkgs/ ), create the directory for your application, and move your default.nix there.
+# If you used <code>with import <nixpkgs> {};</code> to iterate more quickly, now is the time to replace that with the actual dependencies as an attribute set at the beginning of the file e.g. <code>{ lib, stdenv, fetchFromGitHub }:</code>
+# Add the package to the top level declaration of packages. Most of the time this will be https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix .
+# If this is your first package in nixpkgs, add yourself in https://github.com/NixOS/nixpkgs/blob/master/maintainers/maintainer-list.nix
+# Read on about the final steps of branching and sending your PR in https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md .
 == How to install from the local repository ==