NixOS Wiki

Kernel Debugging with QEMU: Difference between revisions

← Older edit
VisualWikitext
imported>Luis-Hebendanz
mNo edit summary
Raboof (talk | contribs)
22 edits
link to PR that fixes building the gdb integration script
 
(27 intermediate revisions by 6 users not shown)
Line 1: Line 1:
== Setup ==
== Set up the QEMU VM ==
 
=== With a Nix kernel ===
Use a NixOS config like this:
 
<syntaxhighlight lang="nix">
{ config, pkgs, lib, modulesPath, ... }:
 
{
  imports = [
    (modulesPath + "/virtualisation/qemu-vm.nix")
  ];
 
  boot.kernelPackages = pkgs.linuxPackagesFor (pkgs.linux.overrideAttrs(a: {
    # To make sure debug_info is not stripped from kernel modules
    dontStrip = true;
  }));
 
  boot.kernelPatches = [
    {
      name = "enable debugging information";
      patch = null;
      extraConfig = ''
        GDB_SCRIPTS y
        DEBUG_INFO y
        KALLSYMS y
      '';
    }
    {
      # https://lkml.kernel.org/r/20250618134629.25700-2-johannes@sipsolutions.net
      name = "scripts: gdb: move MNT_* constants to gdb-parsed";
      patch = (pkgs.fetchpatch {
        url = "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/patch/?id=41a7f737685eed2700654720d3faaffdf0132135";
        hash = "sha256-cWlM6hLGhHUQI87jDbAvLdUb+T34Am+hpjda35Myxfw=";
      });
    }
  ];
  boot.kernelParams = [
    # Avoid kernel address space layout randomization
    "nokaslr"
  ];
  virtualisation.qemu.options = [
    # make qemu listen for gdb on :1234
    "-s"
  ];
 
  environment.systemPackages = with pkgs; [
    vim
    iotop
    gdb
  ];
 
  system.stateVersion = "25.05";
}
</syntaxhighlight>
 
==== Loading module symbols and accurate kernel sources ====
 
Make sure you're using a branch of nixpkgs that has https://github.com/NixOS/nixpkgs/pull/432405 applied.
 
This generates a script that starts gdb with module symbols and kernel sources attached:
 
<syntaxhighlight lang="nix">
  users.motd =
    let
      gdbScript = pkgs.writeScript "attach-gdb.sh" ''
        mkdir -p /tmp/gdb
        cd /tmp/gdb
        rm -rf *
 
        ls ${pkgs.srcOnly config.boot.kernelPackages.kernel} | while read line ; do ln -s ${pkgs.srcOnly config.boot.kernelPackages.kernel}/$line $line ; done
 
        mkdir kos
        cd kos
 
        cp ${config.boot.kernelPackages.kernel}/lib/modules/*/kernel/fs/netfs/* .
        cp ${config.boot.kernelPackages.kernel}/lib/modules/*/kernel/fs/9p/* .
        cp ${config.boot.kernelPackages.kernel}/lib/modules/*/kernel/net/9p/* .
        xz -d *
        GDB_SCRIPT_DIR=$(echo ${config.boot.kernelPackages.kernel.dev}/lib/modules/*/build/scripts/gdb)
 
        gdb \
          -ex "python import sys; sys.path.insert(0, '$GDB_SCRIPT_DIR')" \
          -ex "target remote :1234" \
          -ex "source $GDB_SCRIPT_DIR/vmlinux-gdb.py" \
          -ex "lx-symbols" \
          ${config.boot.kernelPackages.kernel.dev}/vmlinux
      '';
    in ''
      look at /repro/default.json
 
      kernel at ${config.boot.kernelPackages.kernel}
      kernel dev at ${config.boot.kernelPackages.kernel.dev}
 
      attach gdb with ${gdbScript} on the host
    '';
</syntaxhighlight>
 
=== With a manually-built kernel ===
Clone the repository
Clone the repository
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
Line 6: Line 104:


For kernel dependencies,
For kernel dependencies,
create a <code>shell.nix</code> file in the cloned repo
create a <code>shell.nix</code> file in the cloned repo:
 
<syntaxhighlight lang="nix">
{ pkgs ? import <nixpkgs> {} }:
 
pkgs.stdenv.mkDerivation {
  name = "linux-kernel-build";
  nativeBuildInputs = with pkgs; [
    getopt
    flex
    bison
    gcc
    gnumake
    bc
    pkg-config
    binutils
  ];
  buildInputs = with pkgs; [
    elfutils
    ncurses
    openssl
    zlib
  ];
}
</syntaxhighlight>
 
Older kernel versions might buildFHSUserEnv as they have absolute shebangs:
 
<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
{ pkgs ? import <nixpkgs> {} }:
{ pkgs ? import <nixpkgs> {} }:
Line 12: Line 137:
(pkgs.buildFHSUserEnv {
(pkgs.buildFHSUserEnv {
   name = "linux-kernel-build";
   name = "linux-kernel-build";
   targetPkgs = pkgs: (with pkgs;
   targetPkgs = pkgs: (with pkgs; [
  [
     getopt
     getopt
     flex
     flex
     bison
     bison
     libelf
     elfutils
    binutils
     ncurses.dev
     ncurses.dev
     openssl.dev
     openssl.dev
    zlib.dev
     gcc
     gcc
     gnumake
     gnumake
     bc
     bc
   ]);
   ]);
   runScript = "bash";
   runScript = "bash";
Line 29: Line 154:
</syntaxhighlight>
</syntaxhighlight>


== Generate a config for KVM ==
==== Generate a config for KVM ====
If on <code>make</code> you get asked some questions,
If on <code>make</code> you get asked some questions,
just press enter till you are done, this will select the default answer.
just press enter till you are done, this will select the default answer.
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
$ cd linux
$ cd linux
$ make mrproper # Clears all artifacts, do this especially if you upgrade from a significant old version
$ nix-shell shell.nix
$ nix-shell shell.nix
$ make x86_64_defconfig
$ make defconfig kvm_guest.config
$ make kvmconfig
$ scripts/config --set-val DEBUG_INFO y # For gdb debug symbols
$ scripts/config --set-val DEBUG_INFO y # For gdb debug symbols
$ scripts/config --set-val DEBUG y # All pr_debug messages get printed
$ scripts/config --set-val GDB_SCRIPTS y
$ scripts/config --set-val GDB_SCRIPTS y
$ make -j <number-cpu-cores>
$ scripts/config --set-val DEBUG_DRIVER y # Enable printk messages in drivers
# everything as one command for copy'n'paste
$ scripts/config --set-val DEBUG_INFO y --set-val DEBUG y  --set-val GDB_SCRIPTS y --set-val DEBUG_DRIVER y
# this might ask for further options, just press enter for every question
$ make -j$(nproc)
</syntaxhighlight>
</syntaxhighlight>


== Create a bootable Debian image with replaceable kernel ==
==== Create a bootable NixOS image with no kernel ====
Save this as <code>nixos-image.nix</code>:
 
<syntaxhighlight lang="nix">{ pkgs ? import <nixpkgs> {} }:
import (pkgs.path + "/nixos/lib/make-disk-image.nix") {
  config = (import (pkgs.path + "/nixos/lib/eval-config.nix") {
    inherit (pkgs) system;
    modules = [{
      imports = [ ./nixos-config.nix ];
    }];
  }).config;
  inherit pkgs;
  inherit (pkgs) lib;
  diskSize = 2048;
  partitionTableType = "none";
  # for a different format
  format = "qcow2";
}</syntaxhighlight>
 
Than follows the nixos configuration in a file named <code>nixos-config.nix</code>
 
<syntaxhighlight lang="nix">
{ pkgs, lib, modulesPath, ... }:
 
{
  imports = [
    (modulesPath + "/profiles/qemu-guest.nix")
  ];
  boot.loader.grub.enable = false;
  boot.initrd.enable = false;
  boot.isContainer = true;
  boot.loader.initScript.enable = true;
  ## login with empty password
  users.extraUsers.root.initialHashedPassword = "";
 
  networking.firewall.enable = false;
 
  services.getty.helpLine = ''
    Log in as "root" with an empty password.
    If you are connect via serial console:
    Type Ctrl-a c to switch to the qemu console
    and `quit` to stop the VM.
  '';
 
  services.getty.autologinUser = lib.mkDefault "root";
 
  documentation.doc.enable = false;
  documentation.man.enable = false;
  documentation.nixos.enable = false;
  documentation.info.enable = false;
  programs.bash.enableCompletion = false;
  programs.command-not-found.enable = false;
}
</syntaxhighlight>
 
Than build with the following commands:
 
<syntaxhighlight lang="console">
$ nix-build
# copy out
$ install -m644 result/nixos.qcow2 qemu-image.img
</syntaxhighlight>
 
Than follow with the next step is launching qemu.
 
==== Create a bootable Debian image with replaceable kernel ====
If you want to build a different Linux distro you can use the following instructions to build a debian instead:
 
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
  $ nix-shell -p debootstrap qemu
  $ nix-shell -p debootstrap qemu
  $ qemu-img create qemu-image.img
  $ qemu-img create qemu-image.img 5G
  $ mkfs.ext2 qemu-image.img
  $ mkfs.ext2 qemu-image.img
  $ mkdir mount-point.dir
  $ mkdir mount-point.dir
Line 55: Line 252:
  $ exit
  $ exit
  $ sudo umount mount-point.dir
  $ sudo umount mount-point.dir
$ rmdir mount-point.dir
</syntaxhighlight>
</syntaxhighlight>


== Launch qemu ==
==== Installing tools to the image ====
Discard the <code>-enable-kvm</code> flag if
The filesystem is mounted read only so to add tools like lspci. Mount and chroot then use apt to install the needed binaries.
<code> virtualisation.libvirtd.enable </code> is not set in your configuration.nix.
<syntaxhighlight lang="console">
$ sudo  mount -o loop qemu-image.img mount-point.dir
$ sudo chroot mount-point.dir /bin/bash -i
$ export PATH=$PATH:/bin
$ apt install pciutils tree
$ sudo umount mount-point.dir
</syntaxhighlight>
 
==== Launch qemu ====
You can find a slighty stripped version of qemu in a package called <code>qemu_kvm</code> (qemu without emulation support for other cpu architectures).
The <code>nokaslr</code> kernel flag is important to be able to set breakpoints in kernel memory.
The <code>nokaslr</code> kernel flag is important to be able to set breakpoints in kernel memory.
You can also skip the <code>-S</code> to not make qemu break on startup and waiting for gdb.
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
  $ qemu-system-x86_64 -s -S \
  $ qemu-system-x86_64 -s -S \
     -kernel ../arch/x86/boot/bzImage \
     -kernel arch/x86/boot/bzImage \
     -hda qemu-img.img \
     -hda qemu-image.img \
     -append "root=/dev/sda console=ttyS0 nokaslr" \
     -append "root=/dev/sda console=ttyS0 nokaslr" \
     -enable-kvm \
     -enable-kvm \
Line 71: Line 278:
</syntaxhighlight>
</syntaxhighlight>


== Connect with gdb ==
==== Connect with gdb ====
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
  $ echo "add-auto-load-safe-path `pwd`/scripts/gdb/vmlinux-gdb.py" >> ~/.gdbinit
  $ echo "add-auto-load-safe-path `pwd`/scripts/gdb/vmlinux-gdb.py" >> ~/.gdbinit
  $ gdb ./vmlinux
  $ gdb -ex "target remote :1234" ./vmlinux
(gdb) target remote :1234
  (gdb) continue
  (gdb) continue
</syntaxhighlight>
</syntaxhighlight>


== Installing tools to the image ==
Note that setting breakpoints in early boot might not work for all functions.
The filesystem is mounted read only so to add tools like lspci, mount and chroot then use apt to install the needed binaries.
If a breakpoint is not triggered as expected try to set the breakpoint later when the VM
is fully booted.
 
==== Language server support ====
If you want language server support for the kernel code you can generate a compile_commands.json with
<syntaxhighlight lang="console">
$ python ./scripts/clang-tools/gen_compile_commands.py
</syntaxhighlight>
 
This can be used for example in combination with clangd, which scales well to size of the linux kernel.
 
==== Debugging drivers ====
Make sure the driver you want to inspect is not compiled into the kernel, look for the option to enable compilation of your driver, to do this execute:
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
  $ sudo  mount -o loop qemu-image.img mount-point.dir
  $ make nconfig
$ sudo chroot mount-point.dir /bin/bash -i
</syntaxhighlight>
$ export PATH=$PATH:/bin
press <code>F8</code> and search for your driver, and check if it is set to "Module" with <code><M></code>. After compilation copy the driver.ko into the mounted <code>qemu-image.img</code>. Unmount start the kernel and break at the <code>load_module</code> function and <code>insmod driver.ko</code>. Happy hacking!
$ apt install pciutils tree
 
==== Bugs ====
1. With the nixos-config provided above, the console does not work properly. boot.isContainer = true; implies console.enable = false; that disables console. The following can be used as a workaround.
 
<syntaxhighlight lang="nix">
console.enable = true;
systemd.services."serial-getty@ttyS0".enable = true;
</syntaxhighlight>
</syntaxhighlight>
== Using ktest with NixOS ==
Yellow onion has integrated nixos vms into ktest:
https://github.com/YellowOnion/ktest/commit/73fadcff949236927133141fcba4bfd76df632e7
This integration also allows to use incremental kernel builds for rapid development. Checkout the commit message for details.
[[Category:Virtualization]]
Retrieved from "https://wiki.nixos.org/wiki/Kernel_Debugging_with_QEMU"