How to use
- Copy the installation command and review the source package before installing it in your own agent environment.
- Check the source document for required tools, credentials and supported hosts.
- Describe your input and desired deliverable; review the output against the source material.
npx skills add https://github.com/mohitmishra786/low-level-dev-skills --skill kernel-debuggingBefore you start
- Installation does not include a model subscription, API allowance or permission to access your files.
- Repository stars and installation counts describe activity, not an independent quality or security test.
Original skill document
Rendered from the source repository. Headings, examples, code, tables, links, and referenced images are preserved.
Kernel Debugging
Purpose
Guide agents through debugging the Linux kernel: kgdb over serial or USB, kdb built-in debugger commands, ftrace and trace-cmd, kprobes and kretprobes, dynamic debug (dyndbg), crash dump analysis with crash and makedumpfile, dmesg log levels, and /proc/sys runtime tuning for debugging.
When to Use
- Kernel panic or oops with unclear stack trace
- Live debugging a kernel module or driver with kgdb
- Tracing function calls without recompiling (ftrace, kprobes)
- Enabling verbose driver logging with dyndbg
- Analyzing production crash dumps from kdump
- Tuning kernel runtime parameters for debug sessions
Workflow
1. dmesg and log levels
bash
# Kernel ring buffer
dmesg -T -l err,warn
dmesg -w # follow live
# Log level (0=emerg .. 7=debug)
cat /proc/sys/kernel/printk
# current default minimum boot-default
# 4 4 1 7
# Temporarily increase verbosity
echo 8 > /proc/sys/kernel/printkKernel printk levels: KERN_EMERG through KERN_DEBUG. Driver dev_dbg() requires dyndbg or DEBUG define.
2. kgdb over serial
Kernel config: CONFIG_KGDB, CONFIG_KGDB_SERIAL_CONSOLE.
bash
# Boot parameter (ttyS0, 115200)
kgdboc=ttyS0,115200 kgdbwait
# Or at runtime
echo ttyS0,115200 > /sys/module/kgdboc/parameters/kgdboc
echo g > /proc/sysrq-trigger # break into kgdbGDB host side:
bash
# Connect to remote serial
gdb vmlinux
(gdb) set serial baud 115200
(gdb) target remote /dev/ttyUSB0
(gdb) bt
(gdb) list *panic+0x20USB variant: CONFIG_KGDB_KDB with kgdboc=usb on supported hardware.
3. kdb commands
When CONFIG_KGDB_KDB enabled, kdb provides in-kernel shell:
kdb commands (at SysRq break)
├── bt — stack backtrace
├── lsmod — loaded modules
├── md <addr> — memory display
├── ps — process list
├── id <cpu> — CPU registers
├── go — continue execution
└── cpu <n> — switch CPU contextbash
# Enter kdb
echo k > /proc/sysrq-trigger4. ftrace
bash
# Enable function tracer
echo function > /sys/kernel/debug/tracing/current_tracer
echo 1 > /sys/kernel/debug/tracing/tracing_on
# Filter to specific function
echo schedule > /sys/kernel/debug/tracing/set_ftrace_filter
echo ':*probe_*' >> /sys/kernel/debug/tracing/set_ftrace_notrace
# Capture and read
cat /sys/kernel/debug/tracing/trace_pipe
# Disable
echo 0 > /sys/kernel/debug/tracing/tracing_on
echo nop > /sys/kernel/debug/tracing/current_tracerbash
# trace-cmd (userspace recorder)
trace-cmd record -p function -l schedule,do_page_fault
trace-cmd report
trace-cmd.dat # kernelshark GUI5. kprobes and kretprobes
bash
# Dynamic kprobe via debugfs
echo 'p:myprobe do_sys_open %ax %si' > /sys/kernel/debug/tracing/kprobe_events
echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable
cat /sys/kernel/debug/tracing/trace
# kretprobe
echo 'r:myretprobe do_sys_open $retval' >> /sys/kernel/debug/tracing/kprobe_events
# Cleanup
echo 0 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable
echo '-:myprobe' > /sys/kernel/debug/tracing/kprobe_eventsIn-kernel kprobe (module):
c
#include <linux/kprobes.h>
static struct kprobe kp = {
.symbol_name = "do_sys_open",
.pre_handler = my_pre_handler,
};
static int __init mod_init(void)
{
return register_kprobe(&kp);
}6. Dynamic debug (dyndbg)
bash
# Enable all debug messages for a module
echo 'module mydriver +p' > /sys/kernel/debug/dynamic_debug/control
# Enable specific file:line
echo 'file drivers/i2c/i2c-core.c +p' > /sys/kernel/debug/dynamic_debug/control
# Query current settings
grep mydriver /sys/kernel/debug/dynamic_debug/controlBoot-time: dyndbg="module mydriver +p" on kernel command line.
7. kdump and crash analysis
bash
# Setup kdump (crash kernel reserved memory)
# /etc/default/grub: crashkernel=256M
sudo kdump-config show
# After crash, vmcore in /var/crash/
ls /var/crash/*/vmcore
# Analyze with crash utility
crash /usr/lib/debug/boot/vmlinux-$(uname -r) /var/crash/*/vmcore
crash> bt
crash> log
crash> ps
crash> kmem -i
crash> modbash
# Compress vmcore
makedumpfile -c -d /proc/vmcore /tmp/vmcore.lzo8. /proc/sys runtime tuning for debugging
bash
# Panic on oops (reproduce in VM)
sysctl kernel.panic_on_oops=1
# Soft lockup detection
sysctl kernel.softlockup_panic=1
# Watchdog
sysctl kernel.nmi_watchdog=1
# Slab debugging
# Boot: slub_debug=P,pagealloc9. Debugging decision tree
Kernel issue type?
├── Panic/oops → dmesg; crash vmcore; CONFIG_DEBUG_INFO_BTF
├── Logic bug in driver → dyndbg; ftrace function_graph
├── Performance regression → perf record -g -a; trace-cmd
├── Intermittent → kprobes on suspect path
└── Module crash → kgdb; try_module_get auditCommon Problems
| Symptom | Cause | Fix |
|---|---|---|
| kgdb won't connect | Wrong tty/baud | Match kgdboc to serial adapter |
| ftrace empty trace | tracing_off or nop tracer | echo function > current_tracer; enable tracing_on |
| kprobe registration fails | Missing CONFIG_KPROBES or inlined function | Use tracepoint or static_key |
| dyndbg has no effect | CONFIGDYNAMICDEBUG disabled | Rebuild kernel with option |
| crash can't read vmcore | Wrong vmlinux debug symbols | Install linux-image-$(uname -r)-dbg |
| SysRq doesn't work | kernel.sysrq disabled | echo 1 > /proc/sys/kernel/sysrq |
Related Skills
skills/low-level-programming/linux-kernel-modules— module development being debuggedskills/kernel/device-drivers— driver-specific probe/IRQ issuesskills/kernel/kernel-internals— subsystem knowledge for interpreting tracesskills/debuggers/gdb— GDB commands shared with kgdbskills/profilers/linux-perf— perf for kernel hotspotsskills/observability/ebpf— non-invasive kernel tracing alternative
