mohitmishra786/low-level-dev-skills

kernel-debugging-advanced

Advanced kernel debugging skill for ftrace, trace-cmd, perf, kprobes, kgdb, and crash analysis.

View source

How to use

  1. Copy the installation command and review the source package before installing it in your own agent environment.
  2. Check the source document for required tools, credentials and supported hosts.
  3. Describe your input and desired deliverable; review the output against the source material.
npx skills add https://github.com/mohitmishra786/low-level-dev-skills --skill kernel-debugging-advanced

Before you start

  • Installation does not include a model subscription, API allowance or permission to access your files.
  • Repository stars and installation counts describe activity, not an independent quality or security test.
Original skill document

Rendered from the source repository. Headings, examples, code, tables, links, and referenced images are preserved.

Advanced Kernel Debugging

Purpose

Extend skills/kernel/kernel-debugging with production-grade tracing: ftrace, trace-cmd, kernel perf, kprobes/kretprobes, kgdb, crash dump analysis, and printk discipline. Not merged with kernel-debugging — that skill covers baseline kgdb/dyndbg; this one focuses on trace-cmd, function_graph, and live/post-mortem production workflows.

When to Use

  • Latency spikes in kernel driver without obvious bug
  • Tracing function call graph in kernel
  • Live inspection without recompiling (kprobes, bpf)
  • Post-mortem vmcore after panic

Workflow

1. ftrace function graph

bash
# Enable function graph tracer
echo function_graph > /sys/kernel/debug/tracing/current_tracer
echo my_driver_probe > /sys/kernel/debug/tracing/set_graph_function
echo 1 > /sys/kernel/debug/tracing/tracing_on
# reproduce issue
cat /sys/kernel/debug/tracing/trace
echo 0 > /sys/kernel/debug/tracing/tracing_on

Requires CONFIG_FUNCTION_GRAPH_TRACER and debugfs mounted.

2. trace-cmd record

bash
trace-cmd record -p function_graph -F my_probe
trace-cmd report
trace-cmd stat

Portable capture for sharing with others.

3. perf in kernel context

bash
perf record -a -g -- sleep 10
perf report --stdio
perf probe --add my_driver:probe

See skills/profilers/linux-perf for userspace overlap.

4. kprobes (dynamic)

c
#include <linux/kprobes.h>

static struct kprobe kp = {
    .symbol_name = "do_sys_open",
    .pre_handler = handler,
};
register_kprobe(&kp);

Use sparingly in production; prefer static tracepoints when available.

5. dyndbg and printk discipline

bash
echo 'module mydriver +p' > /sys/kernel/debug/dynamic_debug/control
echo 'file drivers/foo/*.c +p' > /sys/kernel/debug/dynamic_debug/control
c
pr_debug("state=%d\n", s);           /* compile-time optional */
printk_ratelimited(KERN_WARNING "hw fault\n");

6. kgdb / kdb

bash
# kernel cmdline: kgdboc=ttyS0,115200 kgdbwait
echo g > /proc/sysrq-trigger   # break into debugger (when configured)

7. Crash dump (vmcore)

bash
# after kdump capture
crash /usr/lib/debug/vmlinux vmcore
crash> bt
crash> dev -s

8. Agent usage

/kernel-debugging-advanced Trace my_driver_ioctl latency with ftrace function_graph

Common Problems

SymptomCauseFix
Empty traceTracer not enabledtracing_on; check current_tracer
ftrace overheadGraph all functionsFilter with set_graph_function
kprobe failInlined symbolTry nearby symbol or static key
kgdb no connectWrong kgdboc ttyMatch serial/USB gadget
crash no symbolsMissing debuginfoInstall linux-image-dbg / vmlinux

Related Skills

  • skills/kernel/kernel-debugging — baseline kgdb, dyndbg
  • skills/profilers/linux-perf — perf record/report
  • skills/observability/ebpf — BPF kprobe alternative
  • skills/kernel-dev/writing-char-drivers — ioctl trace targets
  • skills/debuggers/core-dumps — userspace core parallels

Related by task

from this repository

More skills

All skills
mohitmishra786
Community

low-power-embedded

Low-power embedded skill for sleep modes and energy optimization. Use when configuring MCU sleep/stop/standby, peripheral clock gating, wake-up sources, or measuring firmware current draw. Activates on queries about WFI, STOP mode, STM32 PWR, nRF sleep, wake-up EXTI, or reducing embedded power consumption.

1h installs
1
GitHub stars
296
Updated
Nov 20
mohitmishra786
Community

core-dumps

Core dump analysis skill for production crash triage. Use when loading core files in GDB or LLDB, enabling core dump generation on Linux/macOS, mapping symbols with debuginfo or debuginfod, or extracting backtraces from crashes without re-running the program. Activates on queries about core files, ulimit, coredumpctl, debuginfod, crash triage, or analyzing segfaults from production binaries.

1h installs
2
GitHub stars
267
Updated
Jun 27
mohitmishra786
Community

custom-allocators

Custom allocator skill for memory allocation strategies. Use when implementing pool/slab/arena allocators, tuning jemalloc/mimalloc, writing Rust GlobalAlloc, or benchmarking allocator performance. Activates on queries about jemalloc, mimalloc, tcmalloc, arena allocator, GlobalAlloc, or fragmentation.

1h installs
2
GitHub stars
267
Updated
Jun 27
mohitmishra786
Community

ebpf

eBPF skill for Linux observability and networking. Use when writing eBPF programs with libbpf or bpftrace, attaching kprobes/tracepoints/XDP hooks, debugging verifier errors, working with eBPF maps, or achieving CO-RE portability across kernel versions. Activates on queries about eBPF, bpftool, bpftrace, XDP programs, libbpf, verifier errors, eBPF maps, or kernel tracing with BPF.

1h installs
2
GitHub stars
267
Updated
Jun 27