Enterprise - Revision history

TakodaS: Added dynamic generation of netrc files section

2025-07-25T13:16:54Z

Added dynamic generation of netrc files section

← Older revision		Revision as of 13:16, 25 July 2025
Line 40:		Line 40:
	The proxy itself can be set via the environment variables <code>HTTP_PROXY</code> and <code>HTTPS_PROXY</code>.		The proxy itself can be set via the environment variables <code>HTTP_PROXY</code> and <code>HTTPS_PROXY</code>.

			== Dynamic generation of netrc files ==
			Sometimes you have to deal with dynamically short-lived tokens that must be generated on the fly. The above options do not cover this, so the best way forward is to use `fetchurl`'s `netrcPhase` option:
			newPkgs = pkgs.extend (
			final: prev: {
			fetchurl =
			args:
			(prev.fetchurl.override {
			inherit (pkgs) cacert; # required to avoid infrec
			})
			(
			args
			// {
			netrcPhase =
			# do stuff here to get credentials
			BAR=\'\'$(dynamic-shell-script)
			cat > netrc <<EOF
			machine foobar
			login FOO
			password \'\'$BAR
			EOF
			'';''
			}
			);
			}
			);
			now any fetch in newPkgs will dynamically generate tokens with `fetchurl`.
	[[Category:Networking]]		[[Category:Networking]]

Pigs: Add category networking

2025-05-18T18:40:20Z

Add category networking

← Older revision		Revision as of 18:40, 18 May 2025
Line 1:		Line 1:
	When trying to use Nix and NixOS in corporations there are a number of issues one will run into. This page tries to provide a solution to each of these issues.		When trying to use Nix and NixOS in corporations there are a number of issues one will run into normally because of networking restrictions. This page tries to provide a solution to each of these issues.

	== Private resources ==		== Private resources ==
Line 39:		Line 39:

	The proxy itself can be set via the environment variables <code>HTTP_PROXY</code> and <code>HTTPS_PROXY</code>.		The proxy itself can be set via the environment variables <code>HTTP_PROXY</code> and <code>HTTPS_PROXY</code>.

			[[Category:Networking]]

Kamillaova: fetchFromGithub -> fetchFromGitHub

2024-08-30T18:20:50Z

fetchFromGithub -> fetchFromGitHub

← Older revision		Revision as of 18:20, 30 August 2024
Line 7:		Line 7:
	=== fetchurl ===		=== fetchurl ===

	<code>fetchurl</code> is used to retrieve HTTP resources, but is also used by <code>~~fetchFromGithub~~</code>. For private resources this will usually result in an error like the following:		<code>fetchurl</code> is used to retrieve HTTP resources, but is also used by <code>fetchFromGitHub</code>. For private resources this will usually result in an error like the following:

	curl: (22) The requested URL returned error: 401 Unauthorized		curl: (22) The requested URL returned error: 401 Unauthorized

imported>Bobvanderlinden at 14:10, 28 June 2021

2021-06-28T14:10:49Z

← Older revision		Revision as of 14:10, 28 June 2021
Line 27:		Line 27:
	<syntaxHighlight lang=nix>		<syntaxHighlight lang=nix>
	mypackage = callPackage <mypackage.nix> {		mypackage = callPackage <mypackage.nix> {
	fetchurl = fetchurlBoot;		fetchurl = stdenv.fetchurlBoot;
	};		};
	</syntaxHighlight>		</syntaxHighlight>

imported>Makefu: Document handling of tls intercepting proxies

2019-12-30T23:31:49Z

Document handling of tls intercepting proxies

← Older revision		Revision as of 23:31, 30 December 2019
Line 34:		Line 34:

	== TLS Intercepting Proxy ==		== TLS Intercepting Proxy ==
			TLS-Intercepting proxies will intercept each and every TLS connection and replace the original certificate with it's own to be able to introspect the traffic. This of course creates validation issues with the "official" ca-certificate project.
	~~As of right now there currently does not seem~~ to be ~~a way~~ to ~~install nix packages via an intercepting proxy which replaces~~ the ~~''original'' TLS certificate~~ with the certificate ~~created by the intercepting proxy, see~~ [https://github.com/NixOS/nix/issues/1896 nix ~~issue~~ #~~1896~~].		Since [https://github.com/NixOS/nix/issues/1896 nix pr #2181] you are able to set your intercepting Proxy certificate via <code>NIX_SSL_CERT_FILE</code> to a file on your system which contains the root and intermediate certificates of your proxy.
			See also [https://nixos.org/nix/manual/#sec-nix-ssl-cert-file the appropriate section in the nix manual]

	The proxy itself can be set via the environment variables <code>HTTP_PROXY</code> and <code>HTTPS_PROXY</code>.		The proxy itself can be set via the environment variables <code>HTTP_PROXY</code> and <code>HTTPS_PROXY</code>.

imported>Bobvanderlinden at 06:28, 23 May 2018

2018-05-23T06:28:54Z

← Older revision		Revision as of 06:28, 23 May 2018
Line 11:		Line 11:
	curl: (22) The requested URL returned error: 401 Unauthorized		curl: (22) The requested URL returned error: 401 Unauthorized

	Nix will not know about your credentials in your home directory, as the builders have no access to those files. However, Nix has a few options borrowed from curl that will help in this situation. A netrc file can be used that holds the credentials for all domains that require authorisation. More information on netrc can be found in the [https://www.gnu.org/software/inetutils/manual/html_node/The-_002enetrc-file.html GNU manual].		Nix will not know about your credentials in your home directory, as the builders have no access to those files. However, Nix has a few options borrowed from <code>curl</code> that will help in this situation. A netrc file can be used that holds the credentials for all domains that require authorisation. More information on netrc can be found in the [https://www.gnu.org/software/inetutils/manual/html_node/The-_002enetrc-file.html GNU manual].
	For our example, we will create the file in <code>/etc/nix/netrc</code>. The contents will look similar to the following:		For our example, we will create the file in <code>/etc/nix/netrc</code>. The contents will look similar to the following:

Line 22:		Line 22:
	netrc-file = /etc/nix/netrc		netrc-file = /etc/nix/netrc

	Lastly, the default way of fetching urls is using curl inside a build sandbox. This is a powerful command, but it will not use (and cannot use) a netrc file that is outside of the build sandbox. ~~Note that~~ we do not want to place the netrc file inside the sandbox, because that could leak private credentials into builds. ~~The Nix package manager itself can also fetch HTTP(S) resources. It can do so using '''~~fetchurlBoot~~'''~~. This is ~~usually~~ used to bootstrap some of the more basic packages like ~~'''~~curl~~'''~~ itself, but it can also be very useful for fetching files outside of the sandbox.		Lastly, the default way of fetching urls is using <code>curl</code> inside a build sandbox. This is a powerful command, but it will not use (and cannot use) a netrc file that is outside of the build sandbox. <code>netrc-file</code> is thus only applicable to fetches being done by Nix itself. In addition, we do not want to place the netrc file inside the sandbox, because that could potentially leak private credentials into builds. <code>fetchurlBoot</code> uses this builtin function and makes sure the call-sign is mostly compatible with the regular <code>fetchurl</code>. This function is used most often to bootstrap some of the more basic packages like <code>curl</code> itself, but it can also be very useful for fetching files outside of the sandbox.

	Since '''fetchurlBoot''' is mostly compatible with '''fetchurl''' we can override '''fetchurl''' where needed:		Since '''fetchurlBoot''' is mostly compatible with '''fetchurl''' we can override '''fetchurl''' where needed:

imported>Bobvanderlinden at 23:11, 22 May 2018

2018-05-22T23:11:01Z

← Older revision		Revision as of 23:11, 22 May 2018
Line 24:		Line 24:
	Lastly, the default way of fetching urls is using curl inside a build sandbox. This is a powerful command, but it will not use (and cannot use) a netrc file that is outside of the build sandbox. Note that we do not want to place the netrc file inside the sandbox, because that could leak private credentials into builds. The Nix package manager itself can also fetch HTTP(S) resources. It can do so using '''fetchurlBoot'''. This is usually used to bootstrap some of the more basic packages like '''curl''' itself, but it can also be very useful for fetching files outside of the sandbox.		Lastly, the default way of fetching urls is using curl inside a build sandbox. This is a powerful command, but it will not use (and cannot use) a netrc file that is outside of the build sandbox. Note that we do not want to place the netrc file inside the sandbox, because that could leak private credentials into builds. The Nix package manager itself can also fetch HTTP(S) resources. It can do so using '''fetchurlBoot'''. This is usually used to bootstrap some of the more basic packages like '''curl''' itself, but it can also be very useful for fetching files outside of the sandbox.

	Since ~~```~~fetchurlBoot~~```~~ is mostly compatible with ~~```~~fetchurl~~```~~ we can override ~~```~~fetchurl~~```~~ where needed:		Since '''fetchurlBoot''' is mostly compatible with '''fetchurl''' we can override '''fetchurl''' where needed:
	<syntaxHighlight lang=nix>		<syntaxHighlight lang=nix>
	mypackage = callPackage <mypackage.nix> {		mypackage = callPackage <mypackage.nix> {

imported>Bobvanderlinden: use fetchurlBoot for private HTTP(S) resources

2018-05-22T23:10:28Z

use fetchurlBoot for private HTTP(S) resources

← Older revision		Revision as of 23:10, 22 May 2018
Line 20:		Line 20:
	Next the netrc file needs to be accessible in the builds. We will configure Nix to allow access to this file directly from the build sandboxes. Edit your <code>/etc/nix/nix.conf</code> file so that it includes the following lines:		Next the netrc file needs to be accessible in the builds. We will configure Nix to allow access to this file directly from the build sandboxes. Edit your <code>/etc/nix/nix.conf</code> file so that it includes the following lines:

	~~build~~-~~sandbox-paths~~ = /etc/nix/netrc		netrc-file = /etc/nix/netrc

	Lastly, the ~~builds need to know that they need to~~ use ~~the~~ netrc file ~~in <code>fetchurl</code>~~. ~~We will override~~ the ~~definition of <code>fetchurl</code> to include <code>--~~netrc-file ~~/etc/nix/netrc</code> in~~ the ~~curl options used by <code>fetchurl</code>~~. The ~~following shows how this might look in your~~ Nix ~~file~~:		Lastly, the default way of fetching urls is using curl inside a build sandbox. This is a powerful command, but it will not use (and cannot use) a netrc file that is outside of the build sandbox. Note that we do not want to place the netrc file inside the sandbox, because that could leak private credentials into builds. The Nix package manager itself can also fetch HTTP(S) resources. It can do so using '''fetchurlBoot'''. This is usually used to bootstrap some of the more basic packages like '''curl''' itself, but it can also be very useful for fetching files outside of the sandbox.

			Since ```fetchurlBoot``` is mostly compatible with ```fetchurl``` we can override ```fetchurl``` where needed:
	<syntaxHighlight lang=nix>		<syntaxHighlight lang=nix>
	~~let pkgs~~ = ~~import~~ <~~nixpkgs~~> {		mypackage = callPackage <mypackage.nix> {
	~~config = {~~		fetchurl = fetchurlBoot;
	~~packageOverrides = pkgs: rec {~~		};
	~~fetchurlPrivate = opts: pkgs.~~fetchurl ~~(opts // {~~
	~~curlOpts~~ = ~~"${pkgs.lib.optionalString (opts ? curlOpts) "${opts.curlOpts}"} --netrc-file /etc/nix/netrc";~~
	});
	};
	}
	in
	~~...~~
	</syntaxHighlight>		</syntaxHighlight>

	Now '''~~fetchurlPrivate~~''' ~~can be used just like~~ '''~~fetchurl~~''', ~~but~~ will use the netrc file ~~that includes~~ the credentials ~~of your choice~~ for ~~specific domainnames~~.		Now the package is built exactly the same way as before, but resources will be fetched using '''fetchurlBoot'''. '''fetchurlBoot''' will in turn download the resources within Nix itself, which will use the netrc-file and use the right credentials for the domain names that you have defined.

	== TLS Intercepting Proxy ==		== TLS Intercepting Proxy ==

imported>Parthenon at 22:24, 22 May 2018

2018-05-22T22:24:12Z

← Older revision		Revision as of 22:24, 22 May 2018
Line 3:		Line 3:
	== Private resources ==		== Private resources ==

	Building internal projects will require fetching of internal (private) source code and other resources. These resources usually are protected some form of credentials.		Building internal projects will require fetching of internal (private) source code and other resources. These resources usually are protected by some form of credentials.

	=== fetchurl ===		=== fetchurl ===

imported>Bobvanderlinden at 20:33, 22 May 2018

2018-05-22T20:33:51Z

← Older revision		Revision as of 20:33, 22 May 2018
Line 30:		Line 30:
	curlOpts = "${pkgs.lib.optionalString (opts ? curlOpts) "${opts.curlOpts}"} --netrc-file /etc/nix/netrc";		curlOpts = "${pkgs.lib.optionalString (opts ? curlOpts) "${opts.curlOpts}"} --netrc-file /etc/nix/netrc";
	});		});
	~~fetchurl = fetchurlPrivate;~~
	};		};
	}		}
Line 37:		Line 36:
	</syntaxHighlight>		</syntaxHighlight>

	Now ~~all~~ '''fetchurl''' ~~calls~~ will use the ~~specified~~ netrc file ~~with~~ the credentials of your choice.		Now '''fetchurlPrivate''' can be used just like '''fetchurl''', but will use the netrc file that includes the credentials of your choice for specific domainnames.

	== TLS Intercepting Proxy ==		== TLS Intercepting Proxy ==