Kernel Debugging with QEMU

Setup

Clone the repository

$ git clone https://github.com/torvalds/linux.git

For kernel dependencies, create a shell.nix file in the cloned repo

{ pkgs ? import <nixpkgs> {} }:

(pkgs.buildFHSUserEnv {
  name = "linux-kernel-build";
  targetPkgs = pkgs: (with pkgs;  [
    getopt
    flex
    bison
    # our binutils is currently too old (< 2.32) and breaks with our shipped elfutils
    # https://wiki.gentoo.org/wiki/Binutils_2.32_upgrade_notes/elfutils_0.175:_unable_to_initialize_decompress_status_for_section_.debug_info
    (elfutils.overrideAttrs (old: rec {
      pname = "elfutils";
      version = "0.174";
      name = "${pname}-${version}";
      src = fetchurl {
        url = "https://sourceware.org/elfutils/ftp/${version}/${pname}-${version}.tar.bz2";
        sha256 = "12nhr8zrw4sjzrvpf38vl55bq5nm05qkd7nq76as443f0xq7xwnd";
      };
      NIX_CFLAGS_COMPILE = "-Wno-error=missing-attributes";
    }))
    binutils
    ncurses.dev
    openssl.dev
    zlib.dev
    gcc
    gnumake
    bc
  ]);
  runScript = "bash";
}).env

Generate a config for KVM

If on make you get asked some questions, just press enter till you are done, this will select the default answer.

$ cd linux
$ make mrproper # Clears all artifacts
$ nix-shell shell.nix
$ make x86_64_defconfig
$ make kvm_guest.config
$ scripts/config --set-val DEBUG_INFO y # For gdb debug symbols
$ scripts/config --set-val DEBUG y # All pr_debug messages get printed
$ scripts/config --set-val GDB_SCRIPTS y
$ scripts/config --set-val DEBUG_DRIVER y # Enable printk messages in drivers
$ make -j$(nproc)

Create a bootable Debian image with replaceable kernel

 $ nix-shell -p debootstrap qemu
 $ qemu-img create qemu-image.img 5G
 $ mkfs.ext2 qemu-image.img
 $ mkdir mount-point.dir
 $ sudo mount -o loop qemu-image.img mount-point.dir
 $ sudo debootstrap --arch amd64 buster mount-point.dir
 $ sudo chroot mount-point.dir /bin/bash -i
 $ export PATH=$PATH:/bin
 $ passwd # Set root password
 $ exit
 $ sudo umount mount-point.dir

Launch qemu

The nokaslr kernel flag is important to be able to set breakpoints in kernel memory.

 $ qemu-system-x86_64 -s -S \
    -kernel arch/x86/boot/bzImage \
    -hda qemu-image.img \
    -append "root=/dev/sda console=ttyS0 nokaslr" \
    -enable-kvm \
    -nographic

Connect with gdb

 $ echo "add-auto-load-safe-path `pwd`/scripts/gdb/vmlinux-gdb.py" >> ~/.gdbinit
 $ gdb -ex "target remote :1234" ./vmlinux
 (gdb) continue

Note that setting breakpoints in early boot might not work for all functions. If a breakpoint is not triggered as expected try to set the breakpoint later when the VM is fully booted.

Installing tools to the image

The filesystem is mounted read only so to add tools like lspci. Mount and chroot then use apt to install the needed binaries.

 $ sudo  mount -o loop qemu-image.img mount-point.dir
 $ sudo chroot mount-point.dir /bin/bash -i
 $ export PATH=$PATH:/bin
 $ apt install pciutils tree
 $ sudo umount mount-point.dir

Language server support

If you want language server support for the kernel code you can generate a compile_commands.json with

$ python ./scripts/clang-tools/gen_compile_commands.py

Debugging drivers

Make sure the driver you want to inspect is not compiled into the kernel, look for the option to enable compilation of your driver, to do this execute:

 $ make nconfig

press F8 and search for your driver, and check if it is set to "Module" with <M>. After compilation copy the driver.ko into the mounted qemu-image.img. Unmount start the kernel and break at the load_module function and insmod driver.ko. Happy hacking!