How to use
- Copy the installation command and review the source package before installing it in your own agent environment.
- Check the source document for required tools, credentials and supported hosts.
- Describe your input and desired deliverable; review the output against the source material.
npx skills add https://github.com/mohitmishra786/low-level-dev-skills --skill kernel-debugging-advancedBefore you start
- Installation does not include a model subscription, API allowance or permission to access your files.
- Repository stars and installation counts describe activity, not an independent quality or security test.
Rendered from the source repository. Headings, examples, code, tables, links, and referenced images are preserved.
Advanced Kernel Debugging
Purpose
Extend skills/kernel/kernel-debugging with production-grade tracing: ftrace, trace-cmd, kernel perf, kprobes/kretprobes, kgdb, crash dump analysis, and printk discipline. Not merged with kernel-debugging — that skill covers baseline kgdb/dyndbg; this one focuses on trace-cmd, function_graph, and live/post-mortem production workflows.
When to Use
- Latency spikes in kernel driver without obvious bug
- Tracing function call graph in kernel
- Live inspection without recompiling (kprobes, bpf)
- Post-mortem
vmcoreafter panic
Workflow
1. ftrace function graph
# Enable function graph tracer
echo function_graph > /sys/kernel/debug/tracing/current_tracer
echo my_driver_probe > /sys/kernel/debug/tracing/set_graph_function
echo 1 > /sys/kernel/debug/tracing/tracing_on
# reproduce issue
cat /sys/kernel/debug/tracing/trace
echo 0 > /sys/kernel/debug/tracing/tracing_onRequires CONFIG_FUNCTION_GRAPH_TRACER and debugfs mounted.
2. trace-cmd record
trace-cmd record -p function_graph -F my_probe
trace-cmd report
trace-cmd statPortable capture for sharing with others.
3. perf in kernel context
perf record -a -g -- sleep 10
perf report --stdio
perf probe --add my_driver:probeSee skills/profilers/linux-perf for userspace overlap.
4. kprobes (dynamic)
#include <linux/kprobes.h>
static struct kprobe kp = {
.symbol_name = "do_sys_open",
.pre_handler = handler,
};
register_kprobe(&kp);Use sparingly in production; prefer static tracepoints when available.
5. dyndbg and printk discipline
echo 'module mydriver +p' > /sys/kernel/debug/dynamic_debug/control
echo 'file drivers/foo/*.c +p' > /sys/kernel/debug/dynamic_debug/controlpr_debug("state=%d\n", s); /* compile-time optional */
printk_ratelimited(KERN_WARNING "hw fault\n");6. kgdb / kdb
# kernel cmdline: kgdboc=ttyS0,115200 kgdbwait
echo g > /proc/sysrq-trigger # break into debugger (when configured)7. Crash dump (vmcore)
# after kdump capture
crash /usr/lib/debug/vmlinux vmcore
crash> bt
crash> dev -s8. Agent usage
/kernel-debugging-advanced Trace my_driver_ioctl latency with ftrace function_graphCommon Problems
| Symptom | Cause | Fix |
|---|---|---|
| Empty trace | Tracer not enabled | tracing_on; check current_tracer |
| ftrace overhead | Graph all functions | Filter with set_graph_function |
| kprobe fail | Inlined symbol | Try nearby symbol or static key |
| kgdb no connect | Wrong kgdboc tty | Match serial/USB gadget |
| crash no symbols | Missing debuginfo | Install linux-image-dbg / vmlinux |
Related Skills
skills/kernel/kernel-debugging— baseline kgdb, dyndbgskills/profilers/linux-perf— perf record/reportskills/observability/ebpf— BPF kprobe alternativeskills/kernel-dev/writing-char-drivers— ioctl trace targetsskills/debuggers/core-dumps— userspace core parallels



