Kernel Debugging with QEMU

Setup

Clone the repository

$ git clone https://github.com/torvalds/linux.git

For kernel dependencies, create a shell.nix file in the cloned repo

{ pkgs ? import <nixpkgs> {} }:

(pkgs.buildFHSUserEnv {
  name = "linux-kernel-build";
  targetPkgs = pkgs: (with pkgs;
  [
    getopt
    flex
    bison
    libelf
    ncurses.dev
    openssl.dev
    gcc
    gnumake
    bc

  ]);
  runScript = "bash";
}).env

Generate a config for KVM

If on make you get asked some questions, just press enter till you are done, this will select the default answer.

$ cd linux
$ nix-shell shell.nix
$ make x86_64_defconfig
$ make kvmconfig
$ scripts/config --set-val DEBUG_INFO y # For gdb debug symbols
$ scripts/config --set-val GDB_SCRIPTS y
$ make -j <number-cpu-cores>

Create a bootable Debian image with replaceable kernel

 $ nix-shell -p debootstrap qemu
 $ qemu-img create qemu-image.img
 $ mkfs.ext2 qemu-image.img
 $ mkdir mount-point.dir
 $ sudo mount -o loop qemu-image.img mount-point.dir
 $ sudo debootstrap --arch amd64 buster mount-point.dir
 $ sudo chroot mount-point.dir /bin/bash -i
 $ export PATH=$PATH:/bin
 $ passwd # Set root password
 $ exit
 $ sudo umount mount-point.dir
 $ rmdir mount-point.dir

Launch qemu

Discard the -enable-kvm flag if virtualisation.libvirtd.enable is not set in your configuration.nix. The nokaslr kernel flag is important to be able to set breakpoints in kernel memory.

 $ qemu-system-x86_64 -s -S \
    -kernel arch/x86/boot/bzImage \
    -hda qemu-img.img \
    -append "root=/dev/sda console=ttyS0 nokaslr" \
    -enable-kvm \
    -nographic

Connect with gdb

 $ echo "add-auto-load-safe-path `pwd`/scripts/gdb/vmlinux-gdb.py" >> ~/.gdbinit
 $ gdb ./vmlinux
 (gdb) target remote :1234
 (gdb) continue

Installing tools to the image

The filesystem is mounted read only so to add tools like lspci. Mount and chroot then use apt to install the needed binaries.

 $ sudo  mount -o loop qemu-image.img mount-point.dir
 $ sudo chroot mount-point.dir /bin/bash -i
 $ export PATH=$PATH:/bin
 $ apt install pciutils tree

Debugging drivers

Make sure the driver you want to inspect is not compiled into the kernel, look for the option to enable compilation of your driver, to do this execute:

 $ make nconfig

press F8 and search for your driver, and check if it is set to "Module" with <M>. After compilation copy the driver.ko into the mounted qemu-image.img. Unmount start the kernel and break at the load_module function and insmod driver.ko. Happy hacking!