NixOS Wiki

Kernel Debugging with QEMU: Difference between revisions

← Older editNewer edit →
Visual
imported>Luis-Hebendanz
mNo edit summary
Klinger (talk | contribs)
trusted
898 edits
(26 intermediate revisions by 3 users not shown)
Line 6: Line 6:


For kernel dependencies,
For kernel dependencies,
create a <code>shell.nix</code> file in the cloned repo
create a <code>shell.nix</code> file in the cloned repo:
 
<syntaxhighlight lang="nix">
{ pkgs ? import <nixpkgs> {} }:
 
pkgs.stdenv.mkDerivation {
  name = "linux-kernel-build";
  nativeBuildInputs = with pkgs; [
    getopt
    flex
    bison
    gcc
    gnumake
    bc
    pkg-config
    binutils
  ];
  buildInputs = with pkgs; [
    elfutils
    ncurses
    openssl
    zlib
  ];
}
</syntaxhighlight>
 
Older kernel versions might buildFHSUserEnv as they have absolute shebangs:
 
<syntaxhighlight lang="nix">
<syntaxhighlight lang="nix">
{ pkgs ? import <nixpkgs> {} }:
{ pkgs ? import <nixpkgs> {} }:
Line 12: Line 39:
(pkgs.buildFHSUserEnv {
(pkgs.buildFHSUserEnv {
   name = "linux-kernel-build";
   name = "linux-kernel-build";
   targetPkgs = pkgs: (with pkgs;
   targetPkgs = pkgs: (with pkgs; [
  [
     getopt
     getopt
     flex
     flex
     bison
     bison
     libelf
     elfutils
    binutils
     ncurses.dev
     ncurses.dev
     openssl.dev
     openssl.dev
    zlib.dev
     gcc
     gcc
     gnumake
     gnumake
     bc
     bc
   ]);
   ]);
   runScript = "bash";
   runScript = "bash";
Line 34: Line 61:
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
$ cd linux
$ cd linux
$ make mrproper # Clears all artifacts, do this especially if you upgrade from a significant old version
$ nix-shell shell.nix
$ nix-shell shell.nix
$ make x86_64_defconfig
$ make defconfig kvm_guest.config
$ make kvmconfig
$ scripts/config --set-val DEBUG_INFO y # For gdb debug symbols
$ scripts/config --set-val DEBUG_INFO y # For gdb debug symbols
$ scripts/config --set-val DEBUG y # All pr_debug messages get printed
$ scripts/config --set-val GDB_SCRIPTS y
$ scripts/config --set-val GDB_SCRIPTS y
$ make -j <number-cpu-cores>
$ scripts/config --set-val DEBUG_DRIVER y # Enable printk messages in drivers
# everything as one command for copy'n'paste
$ scripts/config --set-val DEBUG_INFO y --set-val DEBUG y  --set-val GDB_SCRIPTS y --set-val DEBUG_DRIVER y
# this might ask for further options, just press enter for every question
$ make -j$(nproc)
</syntaxhighlight>
</syntaxhighlight>


== Create a bootable debian image with replaceable kernel ==
== Create a bootable NixOS image with no kernel ==
 
Save this as <code>nixos-image.nix</code>:
 
<syntaxhighlight lang="nix">
{ pkgs ? import <nixpkgs> {} }:
import (pkgs.path + "/nixos/lib/make-disk-image.nix") {
  config = (import (pkgs.path + "/nixos/lib/eval-config.nix") {
    inherit (pkgs) system;
    modules = [{
      imports = [ ./nixos-config.nix ];
    }];
  }).config;
  inherit pkgs;
  inherit (pkgs) lib;
  diskSize = 1024;
  partitionTableType = "none";
  # for a different format
  format = "qcow2";
}
</syntaxhighlight>
 
Than follows the nixos configuration in a file named <code>nixos-config.nix</code>
 
<syntaxhighlight lang="nix">
{ pkgs, lib, modulesPath, ... }:
 
{
  imports = [
    (modulesPath + "/profiles/qemu-guest.nix")
  ];
  boot.loader.grub.enable = false;
  boot.initrd.enable = false;
  boot.isContainer = true;
  boot.loader.initScript.enable = true;
  ## login with empty password
  users.extraUsers.root.initialHashedPassword = "";
 
  networking.firewall.enable = false;
 
  services.getty.helpLine = ''
    Log in as "root" with an empty password.
    If you are connect via serial console:
    Type Ctrl-a c to switch to the qemu console
    and `quit` to stop the VM.
  '';
 
  services.getty.autologinUser = lib.mkDefault "root";
 
  documentation.doc.enable = false;
  documentation.man.enable = false;
  documentation.nixos.enable = false;
  documentation.info.enable = false;
  programs.bash.enableCompletion = false;
  programs.command-not-found.enable = false;
}
</syntaxhighlight>
 
Than build with the following commands:
 
<syntaxhighlight lang="console">
$ nix-build
# copy out
$ install -m644 result/nixos.qcow2 qemu-image.img
</syntaxhighlight>
 
Than follow with the next step is launching qemu.
 
== Create a bootable Debian image with replaceable kernel ==
 
If you want to build a different Linux distro you can use the following instructions to build a debian instead:
 
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
  $ nix-shell -p debootstrap qemu
  $ nix-shell -p debootstrap qemu
  $ qemu-img create qemu-image.img
  $ qemu-img create qemu-image.img 5G
  $ mkfs.ext2 qemu-image.img
  $ mkfs.ext2 qemu-image.img
  $ mkdir mount-point.dir
  $ mkdir mount-point.dir
Line 55: Line 158:
  $ exit
  $ exit
  $ sudo umount mount-point.dir
  $ sudo umount mount-point.dir
  $ rmdir mount-point.dir
</syntaxhighlight>
 
=== Installing tools to the image ===
The filesystem is mounted read only so to add tools like lspci. Mount and chroot then use apt to install the needed binaries.
<syntaxhighlight lang="console">
  $ sudo  mount -o loop qemu-image.img mount-point.dir
$ sudo chroot mount-point.dir /bin/bash -i
$ export PATH=$PATH:/bin
$ apt install pciutils tree
$ sudo umount mount-point.dir  
</syntaxhighlight>
</syntaxhighlight>


== Launch qemu ==
== Launch qemu ==
Discard the <code>-enable-kvm</code> flag if
You can find a slighty stripped version of qemu in a package called <code>qemu_kvm</code> (qemu without emulation support for other cpu architectures).
<code> virtualisation.libvirtd.enable </code> is not set in your configuration.nix.
The <code>nokaslr</code> kernel flag is important to be able to set breakpoints in kernel memory.
The <code>nokaslr</code> kernel flag is important to be able to set breakpoints in kernel memory.
You can also skip the <code>-S</code> to not make qemu break on startup and waiting for gdb.
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
  $ qemu-system-x86_64 -s -S \
  $ qemu-system-x86_64 -s -S \
     -kernel ../arch/x86_64/boot/bzImage \
     -kernel arch/x86/boot/bzImage \
     -hda qemu-img.img \
     -hda qemu-image.img \
     -append "root=/dev/sda console=ttyS0 nokaslr" \
     -append "root=/dev/sda console=ttyS0 nokaslr" \
     -enable-kvm \
     -enable-kvm \
Line 74: Line 187:
<syntaxhighlight lang="console">
<syntaxhighlight lang="console">
  $ echo "add-auto-load-safe-path `pwd`/scripts/gdb/vmlinux-gdb.py" >> ~/.gdbinit
  $ echo "add-auto-load-safe-path `pwd`/scripts/gdb/vmlinux-gdb.py" >> ~/.gdbinit
  $ gdb ./vmlinux
  $ gdb -ex "target remote :1234" ./vmlinux
(gdb) target remote :1234
  (gdb) continue
  (gdb) continue
</syntaxhighlight>
</syntaxhighlight>
Note that setting breakpoints in early boot might not work for all functions.
If a breakpoint is not triggered as expected try to set the breakpoint later when the VM
is fully booted.
== Language server support ==
If you want language server support for the kernel code you can generate a compile_commands.json with
<syntaxhighlight lang="console">
$ python ./scripts/clang-tools/gen_compile_commands.py
</syntaxhighlight>
This can be used for example in combination with clangd, which scales well to size of the linux kernel.
== Debugging drivers ==
Make sure the driver you want to inspect is not compiled into the kernel, look for the option to enable compilation of your driver, to do this execute:
<syntaxhighlight lang="console">
$ make nconfig
</syntaxhighlight>
press <code>F8</code> and search for your driver, and check if it is set to "Module" with <code><M></code>. After compilation copy the driver.ko into the mounted <code>qemu-image.img</code>. Unmount start the kernel and break at the <code>load_module</code> function and <code>insmod driver.ko</code>. Happy hacking!
== Bugs ==
1. With the nixos-config provided above, the console does not work properly. boot.isContainer = true; implies console.enable = false; that disables console. The following can be used as a workaround.
<syntaxhighlight lang="nix">
console.enable = true;
systemd.services."serial-getty@ttyS0".enable = true;
</syntaxhighlight>
== Using ktest with NixOS ==
Yellow onion has integrated nixos vms into ktest:
https://github.com/YellowOnion/ktest/commit/73fadcff949236927133141fcba4bfd76df632e7
This integration also allows to use incremental kernel builds for rapid development. Checkout the commit message for details.
[[Category:Virtualization]]
Retrieved from "https://wiki.nixos.org/wiki/Kernel_Debugging_with_QEMU"