NixOS Wiki

Kernel Debugging with QEMU: Difference between revisions

← Older editNewer edit →
Visual
imported>Mic92
generalize make command
Klinger (talk | contribs)
trusted
898 edits
(14 intermediate revisions by 3 users not shown)
Line 6: Line 6:


For kernel dependencies,
For kernel dependencies,
create a <code>shell.nix</code> file in the cloned repo
create a <code>shell.nix</code> file in the cloned repo:
 
<syntaxhighlight lang="nix">
{ pkgs ? import <nixpkgs> {} }:
 
pkgs.stdenv.mkDerivation {
  name = "linux-kernel-build";
  nativeBuildInputs = with pkgs; [
    getopt
    flex
    bison
    gcc
    gnumake
    bc
    pkg-config
    binutils
  ];
  buildInputs = with pkgs; [
    elfutils
    ncurses
    openssl
    zlib
  ];
}
</syntaxhighlight>
 
Older kernel versions might buildFHSUserEnv as they have absolute shebangs:
 
<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
{ pkgs ? import <nixpkgs> {} }:
{ pkgs ? import <nixpkgs> {} }:
Line 12: Line 39:
(pkgs.buildFHSUserEnv {
(pkgs.buildFHSUserEnv {
   name = "linux-kernel-build";
   name = "linux-kernel-build";
   targetPkgs = pkgs: (with pkgs;
   targetPkgs = pkgs: (with pkgs; [
  [
     getopt
     getopt
     flex
     flex
     bison
     bison
     libelf
     elfutils
    binutils
     ncurses.dev
     ncurses.dev
     openssl.dev
     openssl.dev
    zlib.dev
     gcc
     gcc
     gnumake
     gnumake
     bc
     bc
   ]);
   ]);
   runScript = "bash";
   runScript = "bash";
Line 34: Line 61:
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
$ cd linux
$ cd linux
$ make mrproper # Clears all artifacts, do this especially if you upgrade from a significant old version
$ nix-shell shell.nix
$ nix-shell shell.nix
$ make x86_64_defconfig
$ make defconfig kvm_guest.config
$ make kvmconfig
$ scripts/config --set-val DEBUG_INFO y # For gdb debug symbols
$ scripts/config --set-val DEBUG_INFO y # For gdb debug symbols
$ scripts/config --set-val DEBUG y # All pr_debug messages get printed
$ scripts/config --set-val GDB_SCRIPTS y
$ scripts/config --set-val GDB_SCRIPTS y
$ scripts/config --set-val DEBUG_DRIVER y # Enable printk messages in drivers
$ scripts/config --set-val DEBUG_DRIVER y # Enable printk messages in drivers
# everything as one command for copy'n'paste
$ scripts/config --set-val DEBUG_INFO y --set-val DEBUG y  --set-val GDB_SCRIPTS y --set-val DEBUG_DRIVER y
# this might ask for further options, just press enter for every question
$ make -j$(nproc)
$ make -j$(nproc)
</syntaxhighlight>
</syntaxhighlight>
== Create a bootable NixOS image with no kernel ==
Save this as <code>nixos-image.nix</code>:
<syntaxhighlight lang="nix">
{ pkgs ? import <nixpkgs> {} }:
import (pkgs.path + "/nixos/lib/make-disk-image.nix") {
  config = (import (pkgs.path + "/nixos/lib/eval-config.nix") {
    inherit (pkgs) system;
    modules = [{
      imports = [ ./nixos-config.nix ];
    }];
  }).config;
  inherit pkgs;
  inherit (pkgs) lib;
  diskSize = 1024;
  partitionTableType = "none";
  # for a different format
  format = "qcow2";
}
</syntaxhighlight>
Than follows the nixos configuration in a file named <code>nixos-config.nix</code>
<syntaxhighlight lang="nix">
{ pkgs, lib, modulesPath, ... }:
{
  imports = [
    (modulesPath + "/profiles/qemu-guest.nix")
  ];
  boot.loader.grub.enable = false;
  boot.initrd.enable = false;
  boot.isContainer = true;
  boot.loader.initScript.enable = true;
  ## login with empty password
  users.extraUsers.root.initialHashedPassword = "";
  networking.firewall.enable = false;
  services.getty.helpLine = ''
    Log in as "root" with an empty password.
    If you are connect via serial console:
    Type Ctrl-a c to switch to the qemu console
    and `quit` to stop the VM.
  '';
  services.getty.autologinUser = lib.mkDefault "root";
  documentation.doc.enable = false;
  documentation.man.enable = false;
  documentation.nixos.enable = false;
  documentation.info.enable = false;
  programs.bash.enableCompletion = false;
  programs.command-not-found.enable = false;
}
</syntaxhighlight>
Than build with the following commands:
<syntaxhighlight lang="console">
$ nix-build
# copy out
$ install -m644 result/nixos.qcow2 qemu-image.img
</syntaxhighlight>
Than follow with the next step is launching qemu.


== Create a bootable Debian image with replaceable kernel ==
== Create a bootable Debian image with replaceable kernel ==
If you want to build a different Linux distro you can use the following instructions to build a debian instead:
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
  $ nix-shell -p debootstrap qemu
  $ nix-shell -p debootstrap qemu
  $ qemu-img create qemu-image.img
  $ qemu-img create qemu-image.img 5G
  $ mkfs.ext2 qemu-image.img
  $ mkfs.ext2 qemu-image.img
  $ mkdir mount-point.dir
  $ mkdir mount-point.dir
Line 56: Line 158:
  $ exit
  $ exit
  $ sudo umount mount-point.dir
  $ sudo umount mount-point.dir
</syntaxhighlight>
=== Installing tools to the image ===
The filesystem is mounted read only so to add tools like lspci. Mount and chroot then use apt to install the needed binaries.
<syntaxhighlight lang="console">
$ sudo  mount -o loop qemu-image.img mount-point.dir
$ sudo chroot mount-point.dir /bin/bash -i
$ export PATH=$PATH:/bin
$ apt install pciutils tree
$ sudo umount mount-point.dir
</syntaxhighlight>
</syntaxhighlight>


== Launch qemu ==
== Launch qemu ==
You can find a slighty stripped version of qemu in a package called <code>qemu_kvm</code> (qemu without emulation support for other cpu architectures).
The <code>nokaslr</code> kernel flag is important to be able to set breakpoints in kernel memory.
The <code>nokaslr</code> kernel flag is important to be able to set breakpoints in kernel memory.
You can also skip the <code>-S</code> to not make qemu break on startup and waiting for gdb.
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
  $ qemu-system-x86_64 -s -S \
  $ qemu-system-x86_64 -s -S \
     -kernel arch/x86/boot/bzImage \
     -kernel arch/x86/boot/bzImage \
     -hda qemu-img.img \
     -hda qemu-image.img \
     -append "root=/dev/sda console=ttyS0 nokaslr" \
     -append "root=/dev/sda console=ttyS0 nokaslr" \
     -enable-kvm \
     -enable-kvm \
Line 72: Line 187:
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
  $ echo "add-auto-load-safe-path `pwd`/scripts/gdb/vmlinux-gdb.py" >> ~/.gdbinit
  $ echo "add-auto-load-safe-path `pwd`/scripts/gdb/vmlinux-gdb.py" >> ~/.gdbinit
  $ gdb ./vmlinux
  $ gdb -ex "target remote :1234" ./vmlinux
(gdb) target remote :1234
  (gdb) continue
  (gdb) continue
</syntaxhighlight>
</syntaxhighlight>


== Installing tools to the image ==
Note that setting breakpoints in early boot might not work for all functions.
The filesystem is mounted read only so to add tools like lspci. Mount and chroot then use apt to install the needed binaries.
If a breakpoint is not triggered as expected try to set the breakpoint later when the VM
<syntaxhighlight lang="console">
is fully booted.
$ sudo  mount -o loop qemu-image.img mount-point.dir
$ sudo chroot mount-point.dir /bin/bash -i
$ export PATH=$PATH:/bin
$ apt install pciutils tree
$ sudo umount mount-point.dir
</syntaxhighlight>


== Language server support ==
== Language server support ==
If you want language server support for the kernel code you can generate a compile_commands.json with
If you want language server support for the kernel code you can generate a compile_commands.json with
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
$ python ./scripts/gen_compile_commands.py  
$ python ./scripts/clang-tools/gen_compile_commands.py  
</syntaxhighlight>
</syntaxhighlight>
This can be used for example in combination with clangd, which scales well to size of the linux kernel.


== Debugging drivers ==
== Debugging drivers ==
Line 99: Line 209:
</syntaxhighlight>
</syntaxhighlight>
press <code>F8</code> and search for your driver, and check if it is set to "Module" with <code><M></code>. After compilation copy the driver.ko into the mounted <code>qemu-image.img</code>. Unmount start the kernel and break at the <code>load_module</code> function and <code>insmod driver.ko</code>. Happy hacking!
press <code>F8</code> and search for your driver, and check if it is set to "Module" with <code><M></code>. After compilation copy the driver.ko into the mounted <code>qemu-image.img</code>. Unmount start the kernel and break at the <code>load_module</code> function and <code>insmod driver.ko</code>. Happy hacking!
== Bugs ==
1. With the nixos-config provided above, the console does not work properly. boot.isContainer = true; implies console.enable = false; that disables console. The following can be used as a workaround.
<syntaxhighlight lang="nix">
console.enable = true;
systemd.services."serial-getty@ttyS0".enable = true;
</syntaxhighlight>
== Using ktest with NixOS ==
Yellow onion has integrated nixos vms into ktest:
https://github.com/YellowOnion/ktest/commit/73fadcff949236927133141fcba4bfd76df632e7
This integration also allows to use incremental kernel builds for rapid development. Checkout the commit message for details.
[[Category:Virtualization]]
Retrieved from "https://wiki.nixos.org/wiki/Kernel_Debugging_with_QEMU"