Using KGDB Remote Debugging for FreeBSD VMs

This guide explains how to debug a running FreeBSD kernel in a VM using kgdb. The default ddb debugger has decent functionality, but unless you somehow know dynamic memory addresses, those capabilities are effectively unusable. Conversely, kdb offers a more robust interface and set of features.

While there is a handbook section on kgdb, it assumes you are running FreeBSD on bare-metal with a physical serial port connected to another serial host, not on modern virtualized infrastructure. In this guide I hope to bridge this gap and make the process I went through easier for the next person. I ran into this problem after trying to set a breakpoint on a variable, only to realize that ddb required me to know exact memory addresses and setting up kgdb for my use-case was not well documented and hope it benefits someone later down the line.

This was written for using FreeBSD 15.0-CURRENT, but I suspect these will work for the foreseeable future.

High Level Process

At a high level, both hosts (target host and debugger host) will run on Qemuon a Linux base system, however the concepts should work on any hypervisor or OS.. Both hosts will be provisioned with a Serial-over-TCP virtual device. The Target host will be put into debug mode and the Debugger host will connect to it with kgdb.

Getting Setup

Target Host Configuration

The target host running the FreeBSD kernel you intend to debug.

Here, we will install FreeBSD in a Qemu. After provisioning the virtual hardware, add or edit a serial port with the following XML.

<serial type="tcp">
    <source mode="bind" host="0.0.0.0" service="31337" tls="no"/>
    <protocol type="raw"/>
    <target type="isa-serial" port="0">
        <model name="isa-serial"/>
    </target>
    <alias name="serial0"/>
</serial>

This will setup a TCP listener on port 12345 to the serial port. Make sure you change 0.0.0.0 to an IP address on the host system.

Next if you are running a non-CURRENT installation, you need to enable KDB in your kernel by adding the following into your kernel configuration.

makeoptions DEBUG=-g
options KDB
options GDB

For whichever version of FreeBSD you are using, make sure you compile and install a kernel so that you generate debugging symbols. See this section of the handbook for details, but it should be the standard:

cd /usr/src
make buildkernel
make installkernel

Next, you will need to configure the serial port to debugging mode, by setting the port flags to 0x80 in /boot/device.hints, as follows:

hint.uart.0.flags="0x80"

After reboot, you can verify if you have successfully configured KDB by running sysctl debug.kdb.available. You should gdb listed such as below:

debug.kdb.available: ddb gdb

And you should be good.

Debugger Host Configuration

The Debugger host is the VM that you will be running kgdb. If you are using native FreeBSD, you may be able to avoid the virtualization aspect of this. After installing FreeBSD, install kgdb by running pkg install gdb.

Next, configure Qemu to add (or update) a serial port to the VM that will function whose endpoint will be a TCP connection to the target host. To do this, update the XML as follows:

<serial type="tcp">
    <source mode="connect" host="203.0.113.2" service="31337" tls="no"/>
    <protocol type="raw"/>
    <target type="isa-serial" port="0">
        <model name="isa-serial"/>
    </target>
    <alias name="serial0"/>
</serial>

Note that after this update, the debugger host will not boot until the Target host is running. This is because Qemu will try to connect to 127.0.0.1:31337 or fail to boot.

Transfer the Target Host Kernel

kgdb needs the kernel symbol file to map memory addresses to symbols, variables and code. To do this, transfer the kernel file /usr/obj/usr/src/amd64.amd64/sys/GENERIC/kernel.debug from the target to the debugging host. You can typically do this through some variant of the following:

basesystem$ scp user@target:/usr/obj/usr/src/amd64.amd64/sys/GENERIC/kernel.debug
basesystem$ scp kernel.debug user@debugger:

This uses the base system as the middle layer, but feel free to transfer directly between the VMs.

Starting the Debugger

If these steps are completed successfully, start up the target host then the debugger in that order.

On the target host, run:

target# sysctl debug.kdb.current=gdb
debug.kdb.current: ddb -> gdb
target# sysctl debug.kdb.enter=1

If successful, the target machine will appear to lock up. Then on the debugging host, run the following:

debugger# kgdb kernel.debug
...
(kgdb) target remote /dev/cuau0
Remote debugging using /dev/cuau0
Reading symbols from /boot/kernel/intpm.ko...
(No debugging symbols found in /boot/kernel/intpm.ko)
Reading symbols from /boot/kernel/smbus.ko...
(No debugging symbols found in /boot/kernel/smbus.ko)
Reading symbols from /boot/kernel/virtio_console.ko...
(No debugging symbols found in /boot/kernel/virtio_console.ko)
Reading symbols from /boot/kernel/uhid.ko...
(No debugging symbols found in /boot/kernel/uhid.ko)
Reading symbols from /boot/kernel/usbhid.ko...
(No debugging symbols found in /boot/kernel/usbhid.ko)
Reading symbols from /boot/kernel/hidbus.ko...
(No debugging symbols found in /boot/kernel/hidbus.ko)
Reading symbols from /boot/kernel/wmt.ko...
(No debugging symbols found in /boot/kernel/wmt.ko)
breakpoint () at /usr/src/sys/amd64/include/cpufunc.h:62
62	/usr/src/sys/amd64/include/cpufunc.h: No such file or directory.
(kgdb)

And it worked!

From here, you can run normal gdb commands. For example, to get a backtrace, run bt as follows:

(kgdb) bt
#0  breakpoint () at /usr/src/sys/amd64/include/cpufunc.h:62
#1  0xffffffff8104ecf0 in kdb_enter (why=0xffffffff81986d13 "sysctl", msg=0xffffffff81a1e252 "sysctl debug.kdb.enter")
    at /usr/src/sys/kern/subr_kdb.c:555
#2  0xffffffff8104fe75 in kdb_sysctl_enter (oidp=0xffffffff81ef5ad0 <sysctl___debug_kdb_enter>, arg1=0x0, arg2=0, req=0xfffffe00a0fcecc0)
    at /usr/src/sys/kern/subr_kdb.c:203
#3  0xffffffff80ff6815 in sysctl_root_handler_locked (oid=0xffffffff81ef5ad0 <sysctl___debug_kdb_enter>, arg1=0x0, arg2=0,
    req=0xfffffe00a0fcecc0, tracker=0xfffffe00a0fcec30) at /usr/src/sys/kern/kern_sysctl.c:201
#4  0xffffffff80ff55a6 in sysctl_root (oidp=0x0, arg1=0x0, arg2=0, req=0xfffffe00a0fcecc0) at /usr/src/sys/kern/kern_sysctl.c:2409
#5  0xffffffff80ff5fcf in userland_sysctl (td=0xfffff80003a26740, name=0xfffffe00a0fceda0, namelen=3, old=0x0, oldlenp=0x0, inkernel=0,
    new=0x4189c9408008, newlen=4, retval=0xfffffe00a0fced98, flags=0) at /usr/src/sys/kern/kern_sysctl.c:2566
#6  0xffffffff80ff5c75 in sys___sysctl (td=0xfffff80003a26740, uap=0xfffff80003a26b40) at /usr/src/sys/kern/kern_sysctl.c:2439
#7  0xffffffff817b1864 in syscallenter (td=0xfffff80003a26740) at /usr/src/sys/amd64/amd64/../../kern/subr_syscall.c:188
#8  0xffffffff817b0f95 in amd64_syscall (td=0xfffff80003a26740, traced=0) at /usr/src/sys/amd64/amd64/trap.c:1194
#9  <signal handler called>
#10 0x000026a94b0bb89a in ?? ()
Backtrace stopped: Cannot access memory at address 0x26a948d01aa8

Or to switch frames and list the local variables:

(kgdb) frame 8
#8  0xffffffff817b0f95 in amd64_syscall (td=0xfffff80003a26740, traced=0) at /usr/src/sys/amd64/amd64/trap.c:1194
1194	/usr/src/sys/amd64/amd64/trap.c: No such file or directory.
(kgdb) info locals
ksi = {ksi_link = {tqe_next = 0x400000000000, tqe_prev = 0xfffff80003a26740}, ksi_info = {si_signo = -1594036464, si_errno = -512,
    si_code = -2130219481, si_pid = -1, si_uid = 2700930880, si_status = 33553920, si_addr = 0x1f80003a26740, si_value = {
      sival_int = -1594036416, sival_ptr = 0xfffffe00a0fcef40, sigval_int = -1594036416, sigval_ptr = 0xfffffe00a0fcef40}, _reason = {
      _fault = {_trapno = 1}, _timer = {_timerid = 1, _overrun = 0}, _mesgq = {_mqd = 1}, _poll = {_band = 1}, _capsicum = {_syscall = 1},
      __spare__ = {__spare1__ = 1, __spare2__ = {4096, 0, 0, 0, 4096, 0, 0}}}}, ksi_flags = -1594036432,
  ksi_sigq = 0xffffffff817b0748 <trap_check+72>}

Closing Comments

I hope this guide serves to help someone else running into this problem and overall helps in the continued development of FreeBSD. If you have any suggestions on how to improve this guide, please email me at farhan [at] farhan [dot] codes.

Thank you for reading and Happy Hacking!

🇵🇸✌️