Bisecting: Difference between revisions

(8 intermediate revisions by 2 users not shown)

Line 1:

[https://git-scm.com/docs/git-bisect Bisecting] is a feature of version control systems such as [[Git]] and [[Mercurial]] to easily pinpoint regressions. Owing to their reproducibility, Nix and NixOS are well-suited to this. As a result, ~~there are~~ a few ~~common use-cases~~ for Nix.

[https://git-scm.com/docs/git-bisect Bisecting] is a feature of version control systems such as [[Git]] and [[Mercurial]] to easily pinpoint regressions. Owing to their reproducibility, Nix and NixOS are well-suited to this. As a result, we will list a few tips for using tools like this in the Nix context.

== ~~nixpkgs~~ ==

== Commit selection ==

~~either~~:

There are different ways to tweak <code>git bisect</code>'s commit selection to reduce the required builds:

* ~~directly bisect, say from the root of the nixpkgs repository run~~ <code>git bisect start ~~--first-parent master nixos-24.05 && git bisect run nix build -f . lolcat~~</code> ~~if package <code>lolcat</code> broke since <code>nixos-24.05</code>. This uses [https://git-scm.com/docs/git-bisect#_bisect_run git bisect run] to run the test until the culprit is found, while the~~ [https://git-scm.com/docs/git-bisect#Documentation/git-bisect.txt---first-parent <code>--first-parent</code>] ~~flag selects~~ merge commits, which ~~in nixpkgs should be cached in Hydra~~.

* <code>git bisect start</code> flag [https://git-scm.com/docs/git-bisect#Documentation/git-bisect.txt---first-parent <code>--first-parent</code>]: select merge commits, which depending on the repository can help for caching as well as commit stability.

* [https://github.com/~~timokau~~/~~nix-bisect nix-bisect]: smartly pick~~ <code>~~bisect bad/skip~~</code> ~~in automated bisects and give nicer outputs~~

* [https://github.com/blitz/hydrasect <code>hydrasect</code>]: select cached commits cached by Hydra

* [https://git.qyliss.net/hydrasect/about/ hydrasect]: ~~prioritize~~ cached commits ~~in nixpkgs bisects - unfortunately still lacks nix package/flake~~

* [https://github.com/symphorien/nixpkgs-staging-bisecter <code>nixpkgs-staging-bisecter</code>]: reduce number of derivations to be built.

** after each checkout run <code>git checkout $(hydrasect-search | head -1)</code>

* [https://github.com/symphorien/nixpkgs-staging-bisecter nixpkgs-staging-bisecter]: ~~like hydrasect but minimize how many~~ derivations ~~you will build even in staging~~

== ~~Flakes~~ ==

== Automating bisects ==

Bisecting changes in a ~~flake~~ itself can ~~just follow~~ the regular bisecting ~~process~~.

* [https://git-scm.com/docs/git-bisect#_bisect_run <code>git bisect run</code>]: runs the selected command until the culprit is found.

Bisecting ~~regressions in~~ [[~~flake~~]]s' inputs ~~typically involves updating~~ the dependency's <code>~~rev~~</code> ~~and~~ <code>~~narHash~~</code> in the <code>~~flake.lock~~</code> ~~file. Using tools~~ like <code>~~nix~~-~~prefetch~~-~~url~~</code>, <code>~~nix~~-~~hash~~</code>, <code>jq<~~/code~~> ~~and~~ <~~code~~>~~sed~~</code>~~, this process~~ [~~https~~:~~//gist.github.com/KiaraGrouwstra/70bf11002032b3c265512f4e17607631#file-bisect-sh can be scripted~~].

* [https://github.com/timokau/nix-bisect <code>nix-bisect</code>]: helps better judge outcome (<code>git bisect</code>'s <code>good</code> vs <code>bad</code> <code>skip</code>) and gives nicer outputs than <code>git bisect run</code>.

=== Bisecting dependencies ===

While for regressions from changes in a Nix project itself we can bisect following the regular process, bisecting regression in dependencies is a bit different in the sense the repository you would build is separate from the repository you are bisecting.

One can do this by running a bisect from a local checkout of the dependency that induced the regression.

==== Bisecting inputs of [[Flakes|Flake]] projects ====

For dependencies managed using Flakes, such a <code>bisect run</code> can be used with Nix Flake commands' flag <code>--override-input</code> to override specific inputs to use the bisected dependency in its inputs.

Using <code>git bisect run</code> from your local checkout of the dependency that caused the regression, the flake directory to run a command for can be overridden for a NixOS configuration, for example, using [[nixos-rebuild]]'s <code>--flake</code> flag. If we can reproduce the issue using a <code>dry-build</code>, this might then look like: <code>git bisect run nixos-rebuild dry-build --override-input $DEPENDENCY_NAME $(pwd) --flake <NIXOS_CONFIG_DIR>#<FLAKE_ATTRIBUTE></code>.

For a regular Nix Flake project, we do not have a command flag like [[nixos-rebuild]]'s <code>--flake</code> to specify the directory of your project. We might then address this by wrapping the Nix command in a script changing directory there, for example using: <code>git bisect run sh -c "cd <PROJECT_DIR>; nix flake check --override-input <DEPENDENCY_NAME> $(pwd)"</code>.

[[Category:Version control]]

@@ Line 1: / Line 1: @@
-[https://git-scm.com/docs/git-bisect Bisecting] is a feature of version control systems such as [[Git]] and [[Mercurial]] to easily pinpoint regressions. Owing to their reproducibility, Nix and NixOS are well-suited to this. As a result, there are a few common use-cases for Nix.
+[https://git-scm.com/docs/git-bisect Bisecting] is a feature of version control systems such as [[Git]] and [[Mercurial]] to easily pinpoint regressions. Owing to their reproducibility, Nix and NixOS are well-suited to this. As a result, we will list a few tips for using tools like this in the Nix context.
-== nixpkgs ==
+== Commit selection ==
-either:
+There are different ways to tweak <code>git bisect</code>'s commit selection to reduce the required builds:
-* directly bisect, say from the root of the nixpkgs repository run <code>git bisect start --first-parent master nixos-24.05 && git bisect run nix build -f . lolcat</code> if package <code>lolcat</code> broke since <code>nixos-24.05</code>. This uses [https://git-scm.com/docs/git-bisect#_bisect_run git bisect run] to run the test until the culprit is found, while the [https://git-scm.com/docs/git-bisect#Documentation/git-bisect.txt---first-parent <code>--first-parent</code>] flag selects merge commits, which in nixpkgs should be cached in Hydra.
+* <code>git bisect start</code> flag [https://git-scm.com/docs/git-bisect#Documentation/git-bisect.txt---first-parent <code>--first-parent</code>]: select merge commits, which depending on the repository can help for caching as well as commit stability.
-* [https://github.com/timokau/nix-bisect nix-bisect]: smartly pick <code>bisect bad/skip</code> in automated bisects and give nicer outputs
+* [https://github.com/blitz/hydrasect <code>hydrasect</code>]: select cached commits cached by Hydra
-* [https://git.qyliss.net/hydrasect/about/ hydrasect]: prioritize cached commits in nixpkgs bisects - unfortunately still lacks nix package/flake
+* [https://github.com/symphorien/nixpkgs-staging-bisecter <code>nixpkgs-staging-bisecter</code>]: reduce number of derivations to be built.
-** after each checkout run <code>git checkout $(hydrasect-search | head -1)</code>
-* [https://github.com/symphorien/nixpkgs-staging-bisecter nixpkgs-staging-bisecter]: like hydrasect but minimize how many derivations you will build even in staging
-== Flakes ==
+== Automating bisects ==
-Bisecting changes in a flake itself can just follow the regular bisecting process.
+* [https://git-scm.com/docs/git-bisect#_bisect_run <code>git bisect run</code>]: runs the selected command until the culprit is found.
-Bisecting regressions in [[flake]]s' inputs typically involves updating the dependency's <code>rev</code> and <code>narHash</code> in the <code>flake.lock</code> file. Using tools like <code>nix-prefetch-url</code>, <code>nix-hash</code>, <code>jq</code> and <code>sed</code>, this process [https://gist.github.com/KiaraGrouwstra/70bf11002032b3c265512f4e17607631#file-bisect-sh can be scripted].
+* [https://github.com/timokau/nix-bisect <code>nix-bisect</code>]: helps better judge outcome (<code>git bisect</code>'s <code>good</code> vs <code>bad</code> <code>skip</code>) and gives nicer outputs than <code>git bisect run</code>.
+=== Bisecting dependencies ===
+While for regressions from changes in a Nix project itself we can bisect following the regular process, bisecting regression in dependencies is a bit different in the sense the repository you would build is separate from the repository you are bisecting.
+One can do this by running a bisect from a local checkout of the dependency that induced the regression.
+==== Bisecting inputs of [[Flakes|Flake]] projects ====
+For dependencies managed using Flakes, such a <code>bisect run</code> can be used with Nix Flake commands' flag <code>--override-input</code> to override specific inputs to use the bisected dependency in its inputs.
+Using <code>git bisect run</code> from your local checkout of the dependency that caused the regression, the flake directory to run a command for can be overridden for a NixOS configuration, for example, using [[nixos-rebuild]]'s <code>--flake</code> flag. If we can reproduce the issue using a <code>dry-build</code>, this might then look like: <code>git bisect run nixos-rebuild dry-build --override-input $DEPENDENCY_NAME $(pwd) --flake <NIXOS_CONFIG_DIR>#<FLAKE_ATTRIBUTE></code>.
+For a regular Nix Flake project, we do not have a command flag like [[nixos-rebuild]]'s <code>--flake</code> to specify the directory of your project. We might then address this by wrapping the Nix command in a script changing directory there, for example using: <code>git bisect run sh -c "cd <PROJECT_DIR>; nix flake check --override-input <DEPENDENCY_NAME> $(pwd)"</code>.
+[[Category:Version control]]