NixOS Wiki

Nixpkgs/Create and debug packages: Difference between revisions

← Older edit
Visual
imported>Tobias.bora
Artturin (talk | contribs)
trusted
23 edits
 
(14 intermediate revisions by 9 users not shown)
Line 28: Line 28:
There are different steps here depending on whether you're building from source or packaging an existing binary.  There are some common steps too.
There are different steps here depending on whether you're building from source or packaging an existing binary.  There are some common steps too.


=== Packages from source code ===
=== Package from source code ===


# Read the repo build instructions. Many times the instructions won't be 100% clear in the README, but there might be some script for CI that will be more precise. This should give you a rough idea of how hard it will be to package it. For example, if it uses a standard, vanilla build (<code>./configure && make && make install</code>), it will probably be easy to package. If on the contrary there are no CI scripts, or no build instructions, or the build instructions are complicated, things could be rough.  
# Read the repo build instructions and CI scripts (for example, on GitHub, these are located in <code>.github/workflows</code>).
# Look in nixpkgs for a similar package to use as reference. Many times you can just copy/paste a similar package, change the <code>src</code> attribute and things will just work. E.g. if you're packaging a project written in Go, find a package for an existing Go application. Each language has its own supporting Nix functions and a more or less standard way of dealing with things. For example [https://nixos.org/manual/nixpkgs/stable/#sec-language-go Go] has <code>buildGoModule</code>. [https://nixos.org/manual/nixpkgs/stable/#rust Rust] has <code>buildRustPackage</code>. [https://nixos.org/manual/nixpkgs/stable/#python Python] has <code>buildPythonApplication</code>. [https://nixos.org/manual/nixpkgs/stable/#node.js Node.js] has <code>node2nix</code>, <code>yarn2nix</code>, etc. The official documentation for these is generally good. These functions will typically do most of the heavy lifting and you probably just have to point them to the repo and they will do the rest. There are also specific functions for wrapping e.g. [https://nixos.org/manual/nixpkgs/stable/#sec-language-gnome GNOME] applications (<code>wrapGAppsHook</code>), or [https://nixos.org/manual/nixpkgs/stable/#sec-language-qt Qt] apps (<code>libsForQt5</code>, <code>wrapQtAppsHook</code>). It pays off to have a look at these functions in the [https://nixos.org/manual/nixpkgs/stable/#chap-language-support language-support table in the NixOS manual].  
# Look in nixpkgs for a package with a similar build process to use as reference. For example, if you're packaging a project written in Go, find a package for an existing Go application. Each language has its own supporting Nix functions and a more or less standard way of dealing with things. For example, [https://nixos.org/manual/nixpkgs/stable/#sec-language-go Go] has <code>buildGoModule</code>. [https://nixos.org/manual/nixpkgs/stable/#rust Rust] has <code>buildRustPackage</code>. [https://nixos.org/manual/nixpkgs/stable/#python Python] has <code>buildPythonApplication</code>. [https://nixos.org/manual/nixpkgs/stable/#node.js Node.js] has <code>node2nix</code>, <code>yarn2nix</code>, etc. There are also specific functions for wrapping e.g. [https://nixos.org/manual/nixpkgs/stable/#sec-language-gnome GNOME] applications (<code>wrapGAppsHook</code>), or [https://nixos.org/manual/nixpkgs/stable/#sec-language-qt Qt] apps (<code>libsForQt5</code>, <code>wrapQtAppsHook</code>). Refer to the [https://nixos.org/manual/nixpkgs/stable/#chap-language-support language support chapter in the nixpkgs manual].  
# If there isn't a specific function for the build/language you'll need to use [https://nixos.org/guides/nix-pills/fundamentals-of-stdenv.html#idm140737319559808 stdenv.mkDerivation] directly but don't despair (yet!). mkDerivation has built-in support for GNU make, CMake, and possibly others too (?) so you probably won't have to spell out every single command necessary to build the project.
# If there isn't a specific builder for the language, use <code>stdenv.mkDerivation</code> directly, which has built-in support for GNU make (and other build systems, provided you add the necessary <code>nativeBuildInputs</code>).
# Figure out at least some dependencies from the project repo. Doesn't have to be perfect, but something to get you started. See if they're available in nixpkgs (<code>nix search some-library</code> or <code>nix-locate lib/somelibrary.so</code>). If any dependency is missing you'll need to package that as well! Common libraries are usually available though.
# Figure out at least some dependencies from the project repo. See if they're available in nixpkgs (<code>nix search some-library</code> or <code>nix-locate --top-level lib/somelibrary.so</code>). If any dependency is missing you'll need to package that as well.
# Prototype / iterate the build with nix-shell. Run <code>nix-shell -p dep1 dep2 dep3...</code> then run the build manually on the command line. Hopefully this will let you quickly identify missing dependencies and maybe some build quirks. Again this doesn't need to be perfect, don't spend too much time on this.
# Create your derivation in <code>default.nix</code> in some empty local directory.
# Write the default.nix file, save it in some empty local directory.
# At the top of the derivation, temporarily add <code>with import <nixpkgs> {};</code>. For now, don't worry too much about declaring every dependency as a parameter<!-- TODO clarify. "parameter of the default build function"? -->, to save time.
# At the top of default.nix add <code>with import <nixpkgs> {};</code>. For now, don't worry too much about declaring every dependency as a parameter<!-- TODO clarify. "parameter of the default build function"? -->, as it will only slow you down. Again, use some similar package as reference.
# Build the package with <code>nix-build</code>. Iterate on tweaking the derivation and rebuilding until it succeeds.
# Build the package with <code>nix build -L</code>. Iterate on tweaking default.nix and running nix build until it builds.
# For large projects with long compile times, you can use <code>nix-shell</code> instead to [https://nixos.org/manual/nixpkgs/stable/#sec-building-stdenv-package-in-nix-shell run the individual phases].
# Instead of nix build, for large projects with long compile times, you can use [[Development environment with nix-shell|nix-shell]] with <code>nix shell -L</code> (same arguments as nix build). Inside the nix-shell, run something like <code>cd $(mktemp -d) && unpackPhase && cd * && configurePhase && buildPhase && checkPhase && installPhase && fixupPhase</code> (these are the phases of [https://nixos.org/guides/nix-pills/fundamentals-of-stdenv.html stdenv.mkDerivation])
# At this stage, you may encounter some build quirks of the project. Compile-time errors will hopefully explain what you're missing. For example [https://github.com/NixOS/nixpkgs/blob/643ce4bd0f057bc0b90f0faebeb83a3b14f01674/pkgs/tools/package-management/micromamba/default.nix#L6-L10 micromamba needs a specialized build of libsolv].
# At this stage, you may encounter some build-quirks of the project. Compile-time errors will hopefully explain what you're missing. For example [https://github.com/NixOS/nixpkgs/blob/643ce4bd0f057bc0b90f0faebeb83a3b14f01674/pkgs/tools/package-management/micromamba/default.nix#L6-L10 micromamba needs a specialized build of libsolv].
# Read on below for further steps.
# Read on below for further steps.


Line 52: Line 51:
=== Both source code packages and binary packages ===
=== Both source code packages and binary packages ===


# Once you have the package building successfully, it's time to start testing the output. Run <code>nix run -L</code>, this will give you a shell where the application executable should be in the PATH. If the executable is not in the PATH, you might need to override <code>installPhase</code>. Try to test as many functions of the application as you can. Most of the time only making sure that the application starts up will not be enough.
# Once you have the package building successfully, test the output. Ensure the build completes using<code>nix-build</code>, then run <code>result/bin/&lt;executableName></code>. Test as much functionality of the application as you can to ensure that it works as intended.
# Now that your package builds and runs, it's time to move it to nixpkgs. Read [https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md nixpkgs/CONTRIBUTING.md], make sure your package is up to the standards e.g. add a suitable [https://nixos.org/manual/nixpkgs/stable/#sec-standard-meta-attributes <code>meta</code> section].
# Now that your package builds and runs, it's time to move it to nixpkgs. Read [https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md nixpkgs/CONTRIBUTING.md], make sure your package is up to the standards e.g. add a suitable [https://nixos.org/manual/nixpkgs/stable/#sec-standard-meta-attributes <code>meta</code> section].
# Git clone https://github.com/NixOS/nixpkgs , figure out the best category / directory for the application (within https://github.com/NixOS/nixpkgs/tree/master/pkgs/ ), create the directory for your application, and move your default.nix there.
# Git clone https://github.com/NixOS/nixpkgs, figure out the best category / directory for the application (within https://github.com/NixOS/nixpkgs/tree/master/pkgs/), create the directory for your application, and move your default.nix there.
# If you used <code>with import <nixpkgs> {};</code> to iterate more quickly, now is the time to replace that with the actual dependencies as an attribute set at the beginning of the file e.g. <code>{ lib, stdenv, fetchFromGitHub }:</code>
# If you used <code>with import <nixpkgs> {};</code> to iterate more quickly, now is the time to replace that with the actual dependencies as an attribute set at the beginning of the file e.g. <code>{ lib, stdenv, fetchFromGitHub }:</code>
# Add the package to the top level declaration of packages. Most of the time this will be https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix .
# Add the package to the top level declaration of packages. Most of the time this will be https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/all-packages.nix .
# If this is your first package in nixpkgs, add yourself in https://github.com/NixOS/nixpkgs/blob/master/maintainers/maintainer-list.nix
# If this is your first package in nixpkgs, add yourself in https://github.com/NixOS/nixpkgs/blob/master/maintainers/maintainer-list.nix in a separate commit.
# Read on about the final steps of branching and sending your PR in https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md .
# Read on about the final steps of branching and sending your PR in https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md .


== How to install from the local repository ==
== How to install from the local repository ==
Line 199: Line 197:
You would have seen the dependencies downloading, but the ''bc-build'' directory remains empty. The build system would next invoke a builder with some arguments. You can obtain the exact name of the builder (usually '''bash''') and the arguments '''args''' of the builder (typically a shell script) by checking the corresponding value in:
You would have seen the dependencies downloading, but the ''bc-build'' directory remains empty. The build system would next invoke a builder with some arguments. You can obtain the exact name of the builder (usually '''bash''') and the arguments '''args''' of the builder (typically a shell script) by checking the corresponding value in:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
$ nix show-derivation $(nix-instantiate .)
$ nix derivation show $(nix-instantiate .)
</syntaxhighlight>
</syntaxhighlight>


Line 218: Line 216:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
$ export out=~/tmpdev/bc-build/out
$ export out=~/tmpdev/bc-build/out
$ source $stdenv/setup
$ set -x # Optional: it prints all commands, can be practical to debug
$ set -x # Optional: it prints all commands, can be practical to debug
$ set +e # Optional: do not quit the shell on simple errors, Ctrl-C,...
$ set +e # Optional: do not quit the shell on simple errors, Ctrl-C,...
Line 225: Line 222:
</syntaxhighlight>
</syntaxhighlight>


To only run some specific phases:
To only run some specific phases, use runPhase:


<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
$ phases="buildPhase checkPhase" genericBuild
# Syntax: runPhase *phase*
$ runPhase unpackPhase
</syntaxhighlight>
</syntaxhighlight>


Line 373: Line 371:


Tip: A git repository can be used for snapshotting attempts at building the package. This also makes it easy to generate patches, should you need to.
Tip: A git repository can be used for snapshotting attempts at building the package. This also makes it easy to generate patches, should you need to.
== Adding custom libraries and dependencies to a package ==
If you are packaging a dependency, such as a library used by applications for them to compile their code, you might have found you'd like to test if the derivation file installs correctly and can be used by other software.
In order to do this, you'll need to make a simple program that references the library, make a derivation for this program, then add the dependency. For example:
Your program to test the library:
<syntaxhighlight lang="nix">
{
  pkgs ? import <nixpkgs> {
    overlays = [
      (final: prev: {
        my-library = prev.callPackage ./my-library.nix { };
      })
    ];
  },
}:
pkgs.callPackage (
  {
    stdenv,
    hello,
    pkg-config,
    my-library,
  }:
  stdenv.mkDerivation {
    pname = "something";
    version = "1";
    strictDeps = true;
    # host/target agnostic programs
    depsBuildBuild = [
      hello
    ];
    # compilers & linkers & dependecy finding programs
    nativeBuildInputs = [
      pkg-config
    ];
    # libraries
    buildInputs = [
      my-library
    ];
  }
) { }
</syntaxhighlight>


== nix channels ==
== nix channels ==
nix channels can be used in parallel with your new local repositories, see its [[install/remove software#nix-channels| nix-channel-documentation]]
nix channels can be used in parallel with your new local repositories, see its [[install/remove software#nix-channels| nix-channel-documentation]]


== Testing Package Updates with nixpkgs-review ==
You can also use [https://github.com/Mic92/nixpkgs-review nixpkgs-review] to compile, review and merge packages and its dependencies. It claims to be faster than nox and provides a nix-shell where you can test the package.
First make sure it is available in your shell:
<syntaxhighlight lang="bash">
nix-shell -p nixpkgs-review
</syntaxhighlight>
You can run nixpkgs-review against uncommitted/staged changes in a cloned nixpkgs repository:
<syntaxhighlight lang="bash">
cd ~/git/nixpkgs
nixpkgs-review wip [--staged]
</syntaxhighlight>
It is also possible to review a specified commit:
<syntaxhighlight lang="bash">
cd ~/git/nixpkgs
nixpkgs-review rev HEAD
</syntaxhighlight>


== Testing Package Updates with Nox ==
If you have already committed your changes and created a pull request, you can use the pr command:
<syntaxhighlight lang="bash">
cd ~/git/nixpkgs
nixpkgs-review pr 5341
</syntaxhighlight>


If you are updating a package's version, you can use nox to make sure all packages that depend on the updated package still compile correctly.
You can post the build result, approve/merge or read the comments of a pull request inside the provided nix-shell:
<syntaxhighlight lang="bash">
# inside the provided shell
nixpkgs-review post-result
nixpkgs-review approve
nixpkgs-review merge
nixpkgs-review comments
</syntaxhighlight>


First make sure it is available in your shell:
If the pr provides a new package you can start it inside the nix-shell using its package name:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
nix-shell -p nox
# inside the provided shell
packagename
</syntaxhighlight>
</syntaxhighlight>


You can run nox against uncommited changes to a nixpkgs repository:
== Formatting Packages with nixfmt ==
It is "good practice" to format packages in a way that following changed will create as minimal diffs as possible. The formatter [https://github.com/NixOS/nixfmt nixfmt] can be used for that.
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
cd ~/.nix-defexpr
nix-shell -p nixfmt-rfc-style --run 'nixfmt path/to/default.nix'
nox-review wip
</syntaxhighlight>
</syntaxhighlight>


If you have already commited your changes and created a pull request, you can use the pr command:
== Testing Packages with nixpkgs-hammering ==
You can test some "good practices" in a package with [https://github.com/jtojnar/nixpkgs-hammering nixpkgs-hammering]. But before applying the recommendations you should read the given [https://github.com/jtojnar/nixpkgs-hammering/tree/main/explanations explanations].
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
nox-review pr 5341
cd ~/git/nixpkgs
nix run -f https://github.com/jtojnar/nixpkgs-hammering/archive/master.tar.gz -c nixpkgs-hammer packagename # if you are using stable nix
nix run github:jtojnar/nixpkgs-hammering packagename # if you are using nix flake
</syntaxhighlight>
</syntaxhighlight>


Retrieved from "https://wiki.nixos.org/wiki/Nixpkgs/Create_and_debug_packages"