Packaging/Tutorial: Difference between revisions

From NixOS Wiki
imported>Fadenb
imported>Jmarmstrong1207
Add tip about the "Create and debug packages" page existing, and that there is a discussion whether to deprecate this one.
(10 intermediate revisions by 8 users not shown)
Line 1: Line 1:
If you are a new Nix adept and want to package some new software, you will benefit from reading existing Nix package descriptions. The more you read, the faster you'll do your package. I've even based a "Generic Algorithm on Doing Packaging Using Existing Knowledge" on this fact.
{{ outdated | This page has duplicate info from [[Create and debug nix packages|with this other wiki page here]]. That page has much more info that might be of more help for you. This page currently has a discussion if it should be deprecated, but at the meantime it still might be useful until a decision is made to migrate over. }}


===== NOTE1: If you have time, [http://nix-cookbook.readthedocs.io/en/latest/nix-pills.html read Nix Pills] first =====
If you are a new Nix adept and want to package some new software, you will benefit from reading existing Nix package descriptions. The more you read, the faster you'll be able to build your package. I've even based a "Generic Algorithm on Doing Packaging Using Existing Knowledge" on this fact.
 
The nixpkgs manual describes the [https://nixos.org/nixpkgs/manual/#chap-quick-start standard workflow to adding packages to nixpkgs]. Be sure to read this one first.
 
 
===== NOTE 0: If you have time, read [https://unix.stackexchange.com/questions/717168/how-to-package-my-software-in-nix-or-write-my-own-package-derivation-for-nixpkgs this tutorial] first =====
===== NOTE1: If you have time, [https://nixos.org/nixos/nix-pills/ read Nix Pills] first =====


===== NOTE2: If you have even more time, [https://nixos.org/nix/manual/#chap-writing-nix-expressions read Nix manual] to get familiar with Nix syntax =====
===== NOTE2: If you have even more time, [https://nixos.org/nix/manual/#chap-writing-nix-expressions read Nix manual] to get familiar with Nix syntax =====
Line 8: Line 14:


<ol style="list-style-type: decimal;">
<ol style="list-style-type: decimal;">
<li><p>Detect package building category. It may be something simple, that requres <code>gcc</code> and <code>make</code> only, or it may be a python library, or it may be a python application, or it may be [https://nixos.org/nixpkgs/manual/#chap-language-support any other LANGUAGE library/application], or it may be Qt applications, or it may be a Gnome application, or it does uses Mono/wine under a hood, or it vendors it's dependencies, or it is closed source/binary redistributable, or it uses CMake/any other alternative build tool, or it actually is a compiler, ... There are lots of such build categories, and it often isn't reflected at hierarchy level in Nixpkgs source tree.</p></li>
<li><p>Detect package building category. It may be something simple, that requires <code>gcc</code> and <code>make</code> only, or it may be a python library, or it may be a python application, or it may be [https://nixos.org/nixpkgs/manual/#chap-language-support any other LANGUAGE library/application], or it may be Qt applications, or it may be a Gnome application, or it does uses Mono/wine under a hood, or it vendors it's dependencies, or it is closed source/binary redistributable, or it uses CMake/any other alternative build tool, or it actually is a compiler, ... There are lots of such build categories, and it often isn't reflected at hierarchy level in Nixpkgs source tree.</p></li>
<li><p>Try <code>grep</code>ping for building category keywords in Nixpkgs source. Find some example packages, review them and copy one of them as a new expression for your package. Too bad if you didn't find anything, then you should read Nix Pills first and skim through several package expressions in Nixpkgs source to be able to write own one.</p></li>
<li><p>Try <code>grep</code>ping for building category keywords in Nixpkgs source. Find some example packages, review them and copy one of them as a new expression for your package. Too bad if you didn't find anything, then you should read Nix Pills first and skim through several package expressions in Nixpkgs source to be able to write your own.</p></li>
<li><p>Build it without changes, just to test it builds</p></li>
<li><p>Build it without changes, just to test if it builds</p></li>
<li><p>Modify source URL and sha256. Don't forget about that sha256 stuff! What I do, I change last symbol of sha256, let the package fail, and then paste correct sha256 again into expression. There are also various prefetch techniques</p></li>
<li><p>Modify source URL and sha256. Don't forget about that sha256 stuff! What I do, I change the last symbol of the sha256, let the package fail, and then paste the correct sha256 again into expression. There are also various prefetch techniques</p></li>
<li><p>Run <code>nix-shell</code> on this expression (for example: <syntaxhighlight lang="bash" inline>nix-shell '<nixpkgs>' -A hello</syntaxhighlight>). Then run in order:</p>
<li><p>Run <code>nix-shell</code> on this expression (for example: <syntaxhighlight lang="bash" inline>nix-shell '<nixpkgs>' -A hello</syntaxhighlight>). Then run in order:</p>
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
$ unpackPhase
$ unpackPhase
$ cd DIR_AFTER_UNPACK
$ cd $sourceRoot # $sourceRoot is created in unpack phase, see https://nixos.org/manual/nixpkgs/stable/#ssec-unpack-phase for details.
$ patchPhase
$ patchPhase
$ configurePhase
$ configurePhase
$ buildPhase</syntaxhighlight>
$ buildPhase</syntaxhighlight>
<p>(visit [https://nixos.org/nixpkgs/manual/#sec-stdenv-phases stdenv-phases] chapter to learn more about phases)</p></li></ol>
<p>(visit [https://nixos.org/nixpkgs/manual/#sec-stdenv-phases stdenv-phases] chapter to learn more about phases)</p>
 
<p>You may need to run <syntaxhighlight lang="console" inline>$ eval "$configurePhase"</syntaxhighlight> sometimes if the build expression overrides that phase.</p>


You may need to run <syntaxhighlight lang="console" inline>$ eval "$configurePhase"</syntaxhighlight> sometimes if build expression overrides that phase.
<p>It most likely will fail.</p>


It most likely will fail.
<p>NOTE: You can use the [https://nixos.org/manual/nixpkgs/stable/#breakpointhook breakpointHook] to attach into the build environment when an error happens.</p>
 
</li></ol>


<ol start="6" style="list-style-type: decimal;">
<ol start="6" style="list-style-type: decimal;">
<li>Try to find failure category. It may be headers not found, library not found, executable not found, compiler not found, wrong build system, empty result, and many many more... But solutions to these problems are a bit more controllable:</li></ol>
<li>Try to find the failure category. It may be headers not found, library not found, executable not found, compiler not found, wrong build system, empty result, and many many more... But solutions to these problems are a bit more controllable:


<ul>
<ol style="list-style-type: lower-alpha;">
<li><ol style="list-style-type: lower-alpha;">
<li><code>buildInput</code> dependency missing</li>
<li><code>buildInput</code> dependency missing</li></ol>
<li>source requires patching</li>
</li>
<li>you should enter a subdir first</li>
<li><ol start="2" style="list-style-type: lower-alpha;">
<li>you should specify correct build parameters</li>
<li>source requires patching</li></ol>
<li><code>patchShebangs</code> should be run first</li>
</li>
<li>....</li>
<li><ol start="3" style="list-style-type: lower-alpha;">
</ol>
<li>you should enter a subdir first</li></ol>
</li>
<li><ol start="4" style="list-style-type: lower-alpha;">
<li>you should specify correct build parameters</li></ol>
</li>
<li><ol start="5" style="list-style-type: lower-alpha;">
<li><code>patchShebangs</code> should be run first</li></ol>
</li>
<li>....</li></ul>


Don't hesitate to learn from other package manager expressions, like AUR.
Don't hesitate to learn from other package manager expressions, like AUR.
</li>


<ol start="7" style="list-style-type: decimal;">
<li><p>Apply a fix, [https://unix.stackexchange.com/questions/191885/quickly-reload-nix-expression-in-nix-shell exit nix-shell and enter it again]. Do steps 6-7 while there are problems.</p></li>
<li><p>Apply a fix, [https://unix.stackexchange.com/questions/191885/quickly-reload-nix-expression-in-nix-shell exit nix-shell and enter it again]. Do steps 6-7 while there are problems.</p></li>
<li><p>After you got <code>buildPhase</code> to pass and produce correct binaries/libraries, time to <code>installPhase</code>. There are two ways: either you do package developement under root and have write access to Nix store inside nix-shell, or you simply run <code>nix-build</code> on you expression, so Nix will perform it's <code>installPhase</code>.</p></li></ol>
<li><p>After you get the <code>buildPhase</code> to pass and produce correct binaries/libraries, it's time for the<code>installPhase</code>. There are two ways: either you do package development under root and have write access to Nix store inside nix-shell, or you simply run <code>nix-build</code> on you expression, so Nix will perform it's <code>installPhase</code>.</p>


In first case, you can run
<p>In the first case, you can run</p>


<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
$ installPhase
$ installPhase
</syntaxhighlight>
</syntaxhighlight>
directly and examine using <code>tree $out</code> your package files. If something is missing or too many, fix the <code>postInstallPhase</code> or <code>preInstallPhase</code> or <code>installPhase</code> in your expression.
 
<p>directly and examine using <code>tree $out</code> in your package files. If something is missing or too many, fix the <code>postInstallPhase</code> or <code>preInstallPhase</code> or <code>installPhase</code> in your expression.</p>
</li></ol>


<ol start="9" style="list-style-type: decimal;">
<ol start="9" style="list-style-type: decimal;">
<li><p>At this point you should have something that is both buildable and packageable. It still may not be runnable. Run the binaries to find out hidden dependencies, or patches required, or wrapper required, or patchelf required, or ... In general, I use Google and search in Nixpkgs issue tracker related problems.</p></li>
<li><p>At this point you should have something that is both buildable and packageable. It still may not be runnable. Run the binaries to find any hidden dependencies, or patches required, or wrapper required, or patchelf required, or ... In general, I use Google and search in Nixpkgs issue tracker related problems.</p></li>
<li><p>After all the modifications, package should be buildable, runnable and should be self-sufficient. You may add <code>meta</code> section, but it isn't strictly required if you don't plan to publish your result.</p></li></ol>
<li><p>After all the modifications, package should be buildable, runnable and should be self-sufficient. You may add <code>meta</code> section, but it isn't strictly required if you don't plan to publish your result.</p></li></ol>

Revision as of 21:54, 28 February 2024

If you are a new Nix adept and want to package some new software, you will benefit from reading existing Nix package descriptions. The more you read, the faster you'll be able to build your package. I've even based a "Generic Algorithm on Doing Packaging Using Existing Knowledge" on this fact.

The nixpkgs manual describes the standard workflow to adding packages to nixpkgs. Be sure to read this one first.


NOTE 0: If you have time, read this tutorial first
NOTE1: If you have time, read Nix Pills first
NOTE2: If you have even more time, read Nix manual to get familiar with Nix syntax
NOTE3: If you definitely have lots of time, then read Nixpkgs manual
  1. Detect package building category. It may be something simple, that requires gcc and make only, or it may be a python library, or it may be a python application, or it may be any other LANGUAGE library/application, or it may be Qt applications, or it may be a Gnome application, or it does uses Mono/wine under a hood, or it vendors it's dependencies, or it is closed source/binary redistributable, or it uses CMake/any other alternative build tool, or it actually is a compiler, ... There are lots of such build categories, and it often isn't reflected at hierarchy level in Nixpkgs source tree.

  2. Try grepping for building category keywords in Nixpkgs source. Find some example packages, review them and copy one of them as a new expression for your package. Too bad if you didn't find anything, then you should read Nix Pills first and skim through several package expressions in Nixpkgs source to be able to write your own.

  3. Build it without changes, just to test if it builds

  4. Modify source URL and sha256. Don't forget about that sha256 stuff! What I do, I change the last symbol of the sha256, let the package fail, and then paste the correct sha256 again into expression. There are also various prefetch techniques

  5. Run nix-shell on this expression (for example: nix-shell '<nixpkgs>' -A hello). Then run in order:

    $ unpackPhase
    $ cd $sourceRoot # $sourceRoot is created in unpack phase, see https://nixos.org/manual/nixpkgs/stable/#ssec-unpack-phase for details.
    $ patchPhase
    $ configurePhase
    $ buildPhase
    

    (visit stdenv-phases chapter to learn more about phases)

    You may need to run $ eval "$configurePhase" sometimes if the build expression overrides that phase.

    It most likely will fail.

    NOTE: You can use the breakpointHook to attach into the build environment when an error happens.

  1. Try to find the failure category. It may be headers not found, library not found, executable not found, compiler not found, wrong build system, empty result, and many many more... But solutions to these problems are a bit more controllable:
    1. buildInput dependency missing
    2. source requires patching
    3. you should enter a subdir first
    4. you should specify correct build parameters
    5. patchShebangs should be run first
    6. ....

    Don't hesitate to learn from other package manager expressions, like AUR.

  2. Apply a fix, exit nix-shell and enter it again. Do steps 6-7 while there are problems.

  3. After you get the buildPhase to pass and produce correct binaries/libraries, it's time for theinstallPhase. There are two ways: either you do package development under root and have write access to Nix store inside nix-shell, or you simply run nix-build on you expression, so Nix will perform it's installPhase.

    In the first case, you can run

    $ installPhase
    

    directly and examine using tree $out in your package files. If something is missing or too many, fix the postInstallPhase or preInstallPhase or installPhase in your expression.

  1. At this point you should have something that is both buildable and packageable. It still may not be runnable. Run the binaries to find any hidden dependencies, or patches required, or wrapper required, or patchelf required, or ... In general, I use Google and search in Nixpkgs issue tracker related problems.

  2. After all the modifications, package should be buildable, runnable and should be self-sufficient. You may add meta section, but it isn't strictly required if you don't plan to publish your result.