Warning: This is a hack that involves modifying the build scripts. tldr; modify /usr/src/sys/conf/kern.pre.mk to change all references of -O2 to -O0.

Have you ever had dtrace(1) on FreeBSD fail to list a probe that should exist in the kernel? This is because Clang will optimize-out some functions. The result is ctfconvert(1) will not generate debugging symbols that dtrace(1) uses to identify probes. I have a quick solution to getting those probes visible to dtrace(1).

In my case, I was trying to instrument on ieee80211_ioctl_get80211, whose sister function ieee80211_ioctl_set80211 has a dtrace(1) probe in the generic FreeBSD 11 and 12 kernels. Both functions are located in /usr/src/sys/net80211/ieee80211_ioctl.c.

My first attempt was to add to /etc/make.conf as follows and recompile the kernel.

CFLAGS+=-O0 and -fno-inline-functions

This failed to produce the dtrace(1) probe. Several other attempts failed and I was getting inconsistent compilation results (Is it me or is ieee80211_ioctl.c compiled with different flags if NO_CLEAN=1 is set?). When I manually compiled the object file by copying the compilation line for the object file and adding -O0 -fno-inline-functions, nm(1) on both the object file and kernel demonstrated that the symbol was present. I installed the kernel, rebooted and it was listed as a dtrace probe. Great!

But as I continued to debug my WiFi driver (oh yeah, I’m very slowly extending rtwn(4)), I found myself rebuilding the kernel several times and frequently rebooting. Why not do this across the entire kernel?

After hacking around, my solution was to modify the build scripts. My solution was to edit /usr/src/sys/conf/kern.pre.mk and modify all optimization level 2 to optimization level 0. The following is my diff(1) on FreeBSD 12.0-CURRENT.

diff --git a/sys/conf/kern.pre.mk b/sys/conf/kern.pre.mk
index c1bbf0d30bf..9a99f1065aa 100644
--- a/sys/conf/kern.pre.mk
+++ b/sys/conf/kern.pre.mk
@@ -57,14 +57,14 @@ CTFFLAGS+=  -g
.if ${MACHINE_CPUARCH} == "powerpc"
_MINUS_O=      -O      # gcc miscompiles some code at -O2
.else
-_MINUS_O=      -O2
+_MINUS_O=      -O0
.endif
.endif
.if ${MACHINE_CPUARCH} == "amd64"
.if ${COMPILER_TYPE} == "clang"
-COPTFLAGS?=-O2 -pipe
+COPTFLAGS?=-O0 -pipe
.else
-COPTFLAGS?=-O2 -frename-registers -pipe
+COPTFLAGS?=-O0 -frename-registers -pipe
.endif
.else
COPTFLAGS?=${_MINUS_O} -pipe

My dtrace -l | wc -l went from 71432 probes to 91420 probes.

A few thoughts:

• This seems like a hack rather than a long-term solution. Either the problem is with the hard-coded optimization flags, or the inability to overwrite them in all places in make.conf.
• Removing optimizations is only something I would do in a non-production kernel, so its as if I have to choose between optimizations for a production kernel or having dtrace probes. But dtrace explicitly markets itself as not impactful on production.
• Using the dtrace pony as your featured image on WordPress does not render properly and must be rotated and modified. Blame Bryan Cantrill.

If you have a better solution, please let me know and I will update the article, but this works for me!