Debug Symbols

Nix packages rarely embed debugging symbols, and since the notion of "installing" a package is somewhat complicated with nix, one cannot "just" install the foo-dev to magically get debug symbols for foo. Here are some explanations on how to get debug symbols with nix, and especially on NixOS.

By default, packages are stripped and all (most?) debug information is irrevocably lost. If you want to debug an application using a library from such a package, there is little you can do to get debug symbols.

Two types of packages can provide debug symbols:

Unstripped packages

To prevent stripping of a derivation, use the option dontStrip = true;. This still compiles with optimisation; to compile with -Og -ggdb in addition to disabling stripping, you can use the function enableDebugging. Let's take the example of socat. If you install socat, then run

$ gdb socat
Reading symbols from socat...(no debugging symbols found)...done.
(gdb) start
(gdb) info shared
From                To                  Syms Read   Shared Object Library
0x00007ffff7dd8f50  0x00007ffff7df5080  Yes (*)     /nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/ld-linux-x86-64.so.2
0x00007ffff7bd2220  0x00007ffff7bd4fec  Yes (*)     /nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/librt.so.1
0x00007ffff79cde80  0x00007ffff79ce705  Yes (*)     /nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/libutil.so.1
0x00007ffff779afa0  0x00007ffff77bb274  Yes (*)     /nix/store/z2zhmrg6jcrn5iq2779mav0nnq4vm2q6-readline-6.3p08/lib/libreadline.so.6
0x00007ffff752b140  0x00007ffff7569c36  Yes (*)     /nix/store/9kr8r78bwk12050ppywfbhg1vrsd6dp8-openssl-1.0.2p/lib/libssl.so.1.0.0
0x00007ffff717d300  0x00007ffff72b893c  Yes (*)     /nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/libc.so.6
0x00007ffff6d66200  0x00007ffff6eabb8f  Yes (*)     /nix/store/9kr8r78bwk12050ppywfbhg1vrsd6dp8-openssl-1.0.2p/lib/libcrypto.so.1.0.0
0x00007ffff6adbbd0  0x00007ffff6ae9641  Yes (*)     /nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/libpthread.so.0
0x00007ffff687fce0  0x00007ffff68b9768  Yes (*)     /nix/store/s2n99784krxl91mfw3cnn9ylbb5fjvkx-ncurses-6.1/lib/libncursesw.so.6
0x00007ffff6663e60  0x00007ffff6664a9e  Yes (*)     /nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/libdl.so.2
(*): Shared library is missing debugging information.

you see that you have debugging symbols for neither socat itself nor for dependent libraries.

Now, build an unstripped socat:

$ nix-build -E 'with import <nixpkgs> {}; enableDebugging socat'

Let's retry with gdb:

$ gdb result/bin/socat
Reading symbols from result/bin/socat...done.
(gdb) start
Temporary breakpoint 1, main (argc=1, argv=0x7fffffffbe38) at socat.c:84
84	socat.c: No such file or directory.
(gdb) info shared
From                To                  Syms Read   Shared Object Library
0x00007ffff7dd8f50  0x00007ffff7df5080  Yes (*)     /nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/ld-linux-x86-64.so.2
0x00007ffff7bd2220  0x00007ffff7bd4fec  Yes (*)     /nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/librt.so.1
0x00007ffff79cde80  0x00007ffff79ce705  Yes (*)     /nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/libutil.so.1
0x00007ffff779afa0  0x00007ffff77bb274  Yes (*)     /nix/store/z2zhmrg6jcrn5iq2779mav0nnq4vm2q6-readline-6.3p08/lib/libreadline.so.6
0x00007ffff752b140  0x00007ffff7569c36  Yes (*)     /nix/store/9kr8r78bwk12050ppywfbhg1vrsd6dp8-openssl-1.0.2p/lib/libssl.so.1.0.0
0x00007ffff717d300  0x00007ffff72b893c  Yes (*)     /nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/libc.so.6
0x00007ffff6d66200  0x00007ffff6eabb8f  Yes (*)     /nix/store/9kr8r78bwk12050ppywfbhg1vrsd6dp8-openssl-1.0.2p/lib/libcrypto.so.1.0.0
0x00007ffff6adbbd0  0x00007ffff6ae9641  Yes (*)     /nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/libpthread.so.0
0x00007ffff687fce0  0x00007ffff68b9768  Yes (*)     /nix/store/s2n99784krxl91mfw3cnn9ylbb5fjvkx-ncurses-6.1/lib/libncursesw.so.6
0x00007ffff6663e60  0x00007ffff6664a9e  Yes (*)     /nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/libdl.so.2
(*): Shared library is missing debugging information.

You have got debugging symbols for socat, but not its dependencies. Also, gdb complains that it did not find the source files of socat. For source files:

• download the tarball: nix-build "<nixpkgs>" -A socat.src
• extract it: tar xvf result
• back in gdb:

(gdb) dir /tmp/socat-1.7.3.2/
Source directories searched: /tmp/socat-1.7.3.2:$cdir:$cwd
(gdb) where
#0  main (argc=1, argv=0x7fffffffbe38) at socat.c:84
(gdb) l
warning: Source file is more recent than executable.
79	#endif
80	
81	bool havelock;
82	
83	
84	int main(int argc, const char *argv[]) {
85	   const char **arg1, *a;
86	   char *mainwaitstring;
87	   char buff[10];
88	   double rto;

Semi-victory !

To provide debug info for dependencies, we would have to recompile them all with enableDebugging which is time consuming and tedious. The only reasonable solution would be to ship debugging information by default, but it would waste a lot of disk space. This leads to the second type of packages.

Packages with a debug output

Some packages are built with separateDebugInfo = true;. The debug symbols will be stripped from the normal output(s) of the derivation, but instead of being discarded they will be put in a special debug output. Since the library does not depend on this output, no disk space is wasted by default.

The openssl package is such a derivation. Imagine you are debugging a live socat and suddenly want debug symbols for openssl. Previous gdb output tells us the version of openssl we are interested in is /nix/store/9kr8r78bwk12050ppywfbhg1vrsd6dp8-openssl-1.0.2p. We can get back to the original derivation:

$ nix-store --query --deriver /nix/store/9kr8r78bwk12050ppywfbhg1vrsd6dp8-openssl-1.0.2p
/nix/store/i12c15d9rnz45as76s0nsa0nypl0fl6z-openssl-1.0.2p.drv

Then we can get the store path for the debug output:

$ nix show-derivation /nix/store/i12c15d9rnz45as76s0nsa0nypl0fl6z-openssl-1.0.2p.drv | jq '.[]|.outputs.debug.path'
"/nix/store/xd69daaly33m7zid6g31glwmml7lk93f-openssl-1.0.2p-debug"

This store path is probably not yet on our disk, so let's download it:

$ nix-store -r "/nix/store/xd69daaly33m7zid6g31glwmml7lk93f-openssl-1.0.2p-debug"

And now we can tell gdb that debug symbols are in the lib/debug subdirectory with set debug-file-directory:

$ gdb result/bin/socat
(gdb) set debug-file-directory /nix/store/xd69daaly33m7zid6g31glwmml7lk93f-openssl-1.0.2p-debug/lib/debug
(gdb) start
(gdb) info shared
From                To                  Syms Read   Shared Object Library
0x00007ffff7dd8f50  0x00007ffff7df5080  Yes (*)     /nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/ld-linux-x86-64.so.2
0x00007ffff7bd2220  0x00007ffff7bd4fec  Yes (*)     /nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/librt.so.1
0x00007ffff79cde80  0x00007ffff79ce705  Yes (*)     /nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/libutil.so.1
0x00007ffff779afa0  0x00007ffff77bb274  Yes (*)     /nix/store/z2zhmrg6jcrn5iq2779mav0nnq4vm2q6-readline-6.3p08/lib/libreadline.so.6
0x00007ffff752b140  0x00007ffff7569c36  Yes         /nix/store/9kr8r78bwk12050ppywfbhg1vrsd6dp8-openssl-1.0.2p/lib/libssl.so.1.0.0
0x00007ffff717d300  0x00007ffff72b893c  Yes (*)     /nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/libc.so.6
0x00007ffff6d66200  0x00007ffff6eabb8f  Yes         /nix/store/9kr8r78bwk12050ppywfbhg1vrsd6dp8-openssl-1.0.2p/lib/libcrypto.so.1.0.0
0x00007ffff6adbbd0  0x00007ffff6ae9641  Yes (*)     /nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/libpthread.so.0
0x00007ffff687fce0  0x00007ffff68b9768  Yes (*)     /nix/store/s2n99784krxl91mfw3cnn9ylbb5fjvkx-ncurses-6.1/lib/libncursesw.so.6
0x00007ffff6663e60  0x00007ffff6664a9e  Yes (*)     /nix/store/fg4yq8i8wd08xg3fy58l6q73cjy8hjr2-glibc-2.27/lib/libdl.so.2
(*): Shared library is missing debugging information.

Victory!

Unfortunately, it seems you must issue set debug-file-directory before the library is loaded. If you already have typed start or if you have attached a live socat with gdb -p, set debug-file-directory won't have any effect. In this case you can export NIX_DEBUG_INFO_DIRS=/nix/store/xd69daaly33m7zid6g31glwmml7lk93f-openssl-1.0.2p-debug/lib/debug before launching gdb.

NixOS

By toggling environment.enableDebugInfo to 'true' in /etc/nixos/configuration.nix, all separate debug info derivations in your systemPackages will have their debug output linked in /run/current-system/sw/lib/debug/ and will be automatically available to gdb. Though note that this will not pick up debug symbols of dependencies – you will need to add the dependencies you want to debug to environment.systemPackages explicitly. If a derivation you are interested in does not have separate debug info enabled, you still have to override it with an overlay for example.

dwarffs

To avoid the need to explicitly list every dependency in environment.systemPackages to have its debug output available, you can use dwarffs. It will create a virtual file system where gdb will be able to look for separate debug symbols for packages on-demand. The downside is that it might increase gdb start up time significantly.

nixseparatedebuginfo

nixseparatedebuginfod is a debuginfod server that can download the relevant debug outputs and source files as needed by debuginfod-capable tools. Compared to dwarffs, it does not require root access, and handles debug outputs of derivations not built by hydra (eg locally or on a custom binary cache) and source files. Gdb is built with support for debuginfod, and valgrind has support if you additionally install the bin output of elfutils.