Nix For Lang Packaging - Revision history

imported>Vater at 00:04, 17 December 2022

2022-12-17T00:04:38Z

← Older revision		Revision as of 00:04, 17 December 2022
Line 1:		Line 1:
	{{Merge\|~~Nix_Expression_Language~~\|Article is essentially an introduction to the nix language}}		{{Merge\|Nix Expression Language\|Article is essentially an introduction to the nix language}}
	= Overview =		= Overview =

imported>Nix: recategorize

2021-09-20T00:29:00Z

recategorize

← Older revision		Revision as of 00:29, 20 September 2021
Line 116:		Line 116:
	* ...		* ...

	[[Category:~~Discussion~~]]		[[Category:Tutorial]]
	[[Category:Development]]		[[Category:Development]]
	[[Category:Nixpkgs]]		[[Category:Nixpkgs]]

imported>Fab: Update naming

2020-12-15T12:49:47Z

Update naming

← Older revision		Revision as of 12:49, 15 December 2020
Line 104:		Line 104:
	Another difficulty is that because the `.git` folder is not guaranteed to have exactly the same content between clones (due to changes to the pack files), it makes it hard to always get the sha256. In most cases the `.git` folder can be removed to avoid these differences but it's not always possible. For example if the repo needs git submodules, or if the build system invokes `git` to find out repo metadata that get included in the build output, then the `.git` folder needs to be kept.		Another difficulty is that because the `.git` folder is not guaranteed to have exactly the same content between clones (due to changes to the pack files), it makes it hard to always get the sha256. In most cases the `.git` folder can be removed to avoid these differences but it's not always possible. For example if the repo needs git submodules, or if the build system invokes `git` to find out repo metadata that get included in the build output, then the `.git` folder needs to be kept.

	In the ~~github~~ case (and probably ~~bitbucket~~, ~~gitlab~~, ...), the service also provides tarballs of specific hashes. Regularly the tarball output is also changed so nixpkgs resorts on hashing the unpacked content instead.		In the GitHub case (and probably Bitbucket, GitLab, ...), the service also provides tarballs of specific hashes. Regularly the tarball output is also changed so nixpkgs resorts on hashing the unpacked content instead.

	All this means that there are many possible variations and it's going to be hard to agree on which-one is the best.		All this means that there are many possible variations and it's going to be hard to agree on which-one is the best.

imported>Makefu at 16:38, 19 March 2018

2018-03-19T16:38:10Z

← Older revision		Revision as of 16:38, 19 March 2018
Line 1:		Line 1:
			{{Merge\|Nix_Expression_Language\|Article is essentially an introduction to the nix language}}
	= Overview =		= Overview =

imported>Nyarly at 20:29, 3 March 2018

2018-03-03T20:29:17Z

← Older revision		Revision as of 20:29, 3 March 2018
Line 114:		Line 114:
	* The $lang build tool could expose it's build plan to help interspersing nix		* The $lang build tool could expose it's build plan to help interspersing nix
	* ...		* ...

			[[Category:Discussion]]
			[[Category:Development]]
			[[Category:Nixpkgs]]

imported>Nyarly at 20:20, 18 December 2017

2017-12-18T20:20:36Z

← Older revision		Revision as of 20:20, 18 December 2017
Line 73:		Line 73:

	A lot of the time, a $lang2nix is provided to translate the dependency set into nix code. In general this becomes easier when a .lock file is provided because the $lang-specific dependency resolution has already been done. It is even easier if all the dependencies have an associated sha256 that matches how nix calculates them.		A lot of the time, a $lang2nix is provided to translate the dependency set into nix code. In general this becomes easier when a .lock file is provided because the $lang-specific dependency resolution has already been done. It is even easier if all the dependencies have an associated sha256 that matches how nix calculates them.

			== Curating a set of $lang dependencies ==

			Some languages have been collected into lists of $lang packages within Nixpkgs. Each package is then pointed at the nixpkgs versions of its dependencies. (This is the Python preferred approach.)

			Pros:
			* Visibility
			* Fewer touchpoints in the face of updates (esp. for security)

			Cons:
			* More work on Nix-side for both library and application maintainers - the process is very manual.

	== Extending stdenv.mkDerivation ==		== Extending stdenv.mkDerivation ==

imported>Samueldr: Fixes most probably a pythonic mistake with ''' → '' / """

2017-12-18T20:08:21Z

Fixes most probably a pythonic mistake with ''' → '' / """

← Older revision		Revision as of 20:08, 18 December 2017
Line 15:		Line 15:
	src = "...";		src = "...";
	buildInputs = [ lang ];		buildInputs = [ lang ];
	buildPhase = '''		buildPhase = ''
	lang build		lang build
	''';		'';
	installPhase = '''		installPhase = ''
	cp buildfolder $out		cp buildfolder $out
	''';		'';
	}		}
	</syntaxhighlight>		</syntaxhighlight>

imported>Samueldr: (syntaxhighlight!)

2017-12-18T20:07:40Z

(syntaxhighlight!)

← Older revision		Revision as of 20:07, 18 December 2017
Line 9:		Line 9:
	The first integration usually looks like this:		The first integration usually looks like this:

	{ stdenv, lang }:		<syntaxhighlight lang=nix>
	stdenv.mkDerivation {		{ stdenv, lang }:
	name = "my-package";		stdenv.mkDerivation {
	src = "...";		name = "my-package";
	buildInputs = [ lang ];		src = "...";
	buildPhase = '''		buildInputs = [ lang ];
	lang build		buildPhase = '''
	''';		lang build
	installPhase = '''		''';
	cp buildfolder $out		installPhase = '''
	''';		cp buildfolder $out
	}		''';
			}
			</syntaxhighlight>

	Where the `lang build` command:		Where the `lang build` command:

imported>Zimbatm: Some thoughts on integrating Nix with $lang package managers

2017-12-18T20:01:42Z

Some thoughts on integrating Nix with $lang package managers

New page

= Overview =

Nixpkgs includes many packages written in many different languages. Each language has their own special needs which means that we end-up re-implementing similar but not exactly the same features for all these systems.

This page tries to summarize where these differences are found. Ideally this will give you an understanding of why this is a hard problem, and hopefully we can also start changing the languages so that they work better with Nix.

== Step zero ==

The first integration usually looks like this:

{ stdenv, lang }:
stdenv.mkDerivation {
name = "my-package";
src = "...";
buildInputs = [ lang ];
buildPhase = '''
lang build
''';
installPhase = '''
cp buildfolder $out
''';
}

Where the `lang build` command:
* fetches all the project dependencies over the network (usually talking to a package registry)
* unpacks the dependencies into the project structure or in a folder under $HOME
* builds the dependencies
* builds the project

The main advantage is that it's easy to use and doesn't require a lot of coordination between nix and $lang.

On the downside:
* the build is impure, the output may vary between runs
* build sandboxing cannot be enabled (`--option use-sandbox true`).
* if any dependencies of the project change, the whole project needs to be rebuilt

== Fetchers ==

Fetchers are special types of derivation which have network access even under sandbox. In exchange they ask for a sha256 which is checked against the $out at build time. We are thus guaranteed of the output content even if the fetcher our sources change.

This is used by some $lang integration to fetch all the project dependencies, either in individual chunks or as a whole. It means that some cooperation is needed for nix to place the dependencies in the right place for the $lang build system to pick them up.

Plus:
* only fetch if the dependency list has changed
* able to cache the build dependencies

Cons:
* fetcher sha256 changes tend to be discovered late since they get cached
* tools don't generally guarantee on-disk format
* cooperation is needed between $lang and nix

== Interspersing nix with $lang build system ==

When a project has N dependencies, if each dependency could be built in isolation, then only those who change would need to be rebuilt. For this to work, the nix code has to figure out the exact dependency set and run each builds into their own derivation. This is not always possible because languages make assumptions about where the packages are installed and packages start reaching into the filesystem for build dependencies in other packages. Or even worse they start mutating other packages.

Pros:
* less rebuild

Cons:
* higher cooperation needed between $lang and nix

== Overrides ==

When a project has a N dependencies, often these dependencies need additional buildInputs. Or the package needs to be patched for one reason or another.

Depending on how the dependencies are being built, some layer of indirection might be useful to inject those changes into the package build system.

A lot of build issues stem from the lack of distinction between build and runtime dependencies. Some languages don't seem to support that for some reason.

== Generating nix from lockfiles ==

A lot of the time, a $lang2nix is provided to translate the dependency set into nix code. In general this becomes easier when a .lock file is provided because the $lang-specific dependency resolution has already been done. It is even easier if all the dependencies have an associated sha256 that matches how nix calculates them.

== Extending stdenv.mkDerivation ==

It's tricky to extend the stdenv.mkDerivation to make it open for extension. TODO: add more on the subject

== Overlays and binary cache ==

It would be nice to explore how nixpkgs-$lang overlays could be used as extensions of the package registries. I played with that with the rubygems.org repo and just the hashes where already taking 1GB on disk. This leads me to believe that we would be better off with API results.

Associated to that, a lot of developers are dreaming of having a $lang binary cache that they can tap into. Especially with languages like Haskell that take a long time to build. Compiling all the packages, times number of languages, times number of build input changes in nixpkgs. This leads me to believe that it would be better to help said developer to setup a CI + binary cache that they can use that only compiles the subset of packages and single nixpkgs snapshot that they are using.

== Git dependencies and hashing ==

One difficulty is to agree on what hash should be generated for git repositories.

Most languages assume that a repo URL and git commit SHA1 are enough to uniquely identify the code. But nix also wants a sha256.

Another difficulty is that because the `.git` folder is not guaranteed to have exactly the same content between clones (due to changes to the pack files), it makes it hard to always get the sha256. In most cases the `.git` folder can be removed to avoid these differences but it's not always possible. For example if the repo needs git submodules, or if the build system invokes `git` to find out repo metadata that get included in the build output, then the `.git` folder needs to be kept.

In the github case (and probably bitbucket, gitlab, ...), the service also provides tarballs of specific hashes. Regularly the tarball output is also changed so nixpkgs resorts on hashing the unpacked content instead.

All this means that there are many possible variations and it's going to be hard to agree on which-one is the best.

== What $lang can do to make our lives easier ==

* The package registry should return sha256 of the packages with all their API results
* Avoid changing the on-disk format of unpacked packages
* Lockfiles should contain sha256 of all the dependencies
* The $lang build tool could expose it's build plan to help interspersing nix
* ...