Kernel Debugging with QEMU: Difference between revisions
m Increase image size as 1024 is no longer sufficient |
document how to debug a nix-built kernel including modules and sources |
||
| Line 1: | Line 1: | ||
== | == Set up the QEMU VM == | ||
=== With a Nix kernel === | |||
Use a NixOS config like this: | |||
<syntaxhighlight lang="nix"> | |||
{ config, pkgs, lib, modulesPath, ... }: | |||
{ | |||
imports = [ | |||
(modulesPath + "/virtualisation/qemu-vm.nix") | |||
]; | |||
boot.kernelPackages = pkgs.linuxPackagesFor (pkgs.linux.overrideAttrs(a: { | |||
# To make sure debug_info is not stripped from kernel modules | |||
dontStrip = true; | |||
})); | |||
boot.kernelPatches = [ | |||
{ | |||
name = "enable debugging information"; | |||
patch = null; | |||
extraConfig = '' | |||
GDB_SCRIPTS y | |||
DEBUG_INFO y | |||
KALLSYMS y | |||
''; | |||
} | |||
{ | |||
# https://lkml.kernel.org/r/20250618134629.25700-2-johannes@sipsolutions.net | |||
name = "scripts: gdb: move MNT_* constants to gdb-parsed"; | |||
patch = (pkgs.fetchpatch { | |||
url = "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/patch/?id=41a7f737685eed2700654720d3faaffdf0132135"; | |||
hash = "sha256-cWlM6hLGhHUQI87jDbAvLdUb+T34Am+hpjda35Myxfw="; | |||
}); | |||
} | |||
]; | |||
boot.kernelParams = [ | |||
# Avoid kernel address space layout randomization | |||
"nokaslr" | |||
]; | |||
virtualisation.qemu.options = [ | |||
# make qemu listen for gdb on :1234 | |||
"-s" | |||
]; | |||
environment.systemPackages = with pkgs; [ | |||
vim | |||
iotop | |||
gdb | |||
]; | |||
system.stateVersion = "25.05"; | |||
} | |||
</syntaxhighlight> | |||
==== Loading module symbols and accurate kernel sources ==== | |||
This generates a script that starts gdb with module symbols and kernel sources attached: | |||
<syntaxhighlight lang="nix"> | |||
users.motd = | |||
let | |||
gdbScript = pkgs.writeScript "attach-gdb.sh" '' | |||
mkdir -p /tmp/gdb | |||
cd /tmp/gdb | |||
rm -rf * | |||
ls ${pkgs.srcOnly config.boot.kernelPackages.kernel} | while read line ; do ln -s ${pkgs.srcOnly config.boot.kernelPackages.kernel}/$line $line ; done | |||
mkdir kos | |||
cd kos | |||
cp ${config.boot.kernelPackages.kernel}/lib/modules/*/kernel/fs/netfs/* . | |||
cp ${config.boot.kernelPackages.kernel}/lib/modules/*/kernel/fs/9p/* . | |||
cp ${config.boot.kernelPackages.kernel}/lib/modules/*/kernel/net/9p/* . | |||
xz -d * | |||
GDB_SCRIPT_DIR=$(echo ${config.boot.kernelPackages.kernel.dev}/lib/modules/*/build/scripts/gdb) | |||
gdb \ | |||
-ex "python import sys; sys.path.insert(0, '$GDB_SCRIPT_DIR')" \ | |||
-ex "target remote :1234" \ | |||
-ex "source $GDB_SCRIPT_DIR/vmlinux-gdb.py" \ | |||
-ex "lx-symbols" \ | |||
${config.boot.kernelPackages.kernel.dev}/vmlinux | |||
''; | |||
in '' | |||
look at /repro/default.json | |||
kernel at ${config.boot.kernelPackages.kernel} | |||
kernel dev at ${config.boot.kernelPackages.kernel.dev} | |||
attach gdb with ${gdbScript} on the host | |||
''; | |||
</syntaxhighlight> | |||
=== With a manually-built kernel === | |||
Clone the repository | Clone the repository | ||
<syntaxhighlight lang="console"> | <syntaxhighlight lang="console"> | ||
| Line 56: | Line 152: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
== Generate a config for KVM == | ==== Generate a config for KVM ==== | ||
If on <code>make</code> you get asked some questions, | If on <code>make</code> you get asked some questions, | ||
just press enter till you are done, this will select the default answer. | just press enter till you are done, this will select the default answer. | ||
| Line 74: | Line 170: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
== Create a bootable NixOS image with no kernel == | ==== Create a bootable NixOS image with no kernel ==== | ||
Save this as <code>nixos-image.nix</code>: | Save this as <code>nixos-image.nix</code>: | ||
| Line 140: | Line 235: | ||
Than follow with the next step is launching qemu. | Than follow with the next step is launching qemu. | ||
== Create a bootable Debian image with replaceable kernel == | ==== Create a bootable Debian image with replaceable kernel ==== | ||
If you want to build a different Linux distro you can use the following instructions to build a debian instead: | If you want to build a different Linux distro you can use the following instructions to build a debian instead: | ||
| Line 158: | Line 252: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
=== Installing tools to the image === | ==== Installing tools to the image ==== | ||
The filesystem is mounted read only so to add tools like lspci. Mount and chroot then use apt to install the needed binaries. | The filesystem is mounted read only so to add tools like lspci. Mount and chroot then use apt to install the needed binaries. | ||
<syntaxhighlight lang="console"> | <syntaxhighlight lang="console"> | ||
| Line 168: | Line 262: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
== Launch qemu == | ==== Launch qemu ==== | ||
You can find a slighty stripped version of qemu in a package called <code>qemu_kvm</code> (qemu without emulation support for other cpu architectures). | You can find a slighty stripped version of qemu in a package called <code>qemu_kvm</code> (qemu without emulation support for other cpu architectures). | ||
The <code>nokaslr</code> kernel flag is important to be able to set breakpoints in kernel memory. | The <code>nokaslr</code> kernel flag is important to be able to set breakpoints in kernel memory. | ||
| Line 182: | Line 276: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
== Connect with gdb == | ==== Connect with gdb ==== | ||
<syntaxhighlight lang="console"> | <syntaxhighlight lang="console"> | ||
$ echo "add-auto-load-safe-path `pwd`/scripts/gdb/vmlinux-gdb.py" >> ~/.gdbinit | $ echo "add-auto-load-safe-path `pwd`/scripts/gdb/vmlinux-gdb.py" >> ~/.gdbinit | ||
| Line 193: | Line 287: | ||
is fully booted. | is fully booted. | ||
== Language server support == | ==== Language server support ==== | ||
If you want language server support for the kernel code you can generate a compile_commands.json with | If you want language server support for the kernel code you can generate a compile_commands.json with | ||
<syntaxhighlight lang="console"> | <syntaxhighlight lang="console"> | ||
| Line 201: | Line 295: | ||
This can be used for example in combination with clangd, which scales well to size of the linux kernel. | This can be used for example in combination with clangd, which scales well to size of the linux kernel. | ||
== Debugging drivers == | ==== Debugging drivers ==== | ||
Make sure the driver you want to inspect is not compiled into the kernel, look for the option to enable compilation of your driver, to do this execute: | Make sure the driver you want to inspect is not compiled into the kernel, look for the option to enable compilation of your driver, to do this execute: | ||
<syntaxhighlight lang="console"> | <syntaxhighlight lang="console"> | ||
| Line 208: | Line 302: | ||
press <code>F8</code> and search for your driver, and check if it is set to "Module" with <code><M></code>. After compilation copy the driver.ko into the mounted <code>qemu-image.img</code>. Unmount start the kernel and break at the <code>load_module</code> function and <code>insmod driver.ko</code>. Happy hacking! | press <code>F8</code> and search for your driver, and check if it is set to "Module" with <code><M></code>. After compilation copy the driver.ko into the mounted <code>qemu-image.img</code>. Unmount start the kernel and break at the <code>load_module</code> function and <code>insmod driver.ko</code>. Happy hacking! | ||
== Bugs == | ==== Bugs ==== | ||
1. With the nixos-config provided above, the console does not work properly. boot.isContainer = true; implies console.enable = false; that disables console. The following can be used as a workaround. | 1. With the nixos-config provided above, the console does not work properly. boot.isContainer = true; implies console.enable = false; that disables console. The following can be used as a workaround. | ||