mohitmishra786/low-level-dev-skills

cuda-debugging

CUDA debugging skill for GPU program correctness.

ソースを見る

使い方

  1. インストールコマンドをコピーし、自分の環境へ導入する前にソースを確認します。
  2. 必要なツール、認証情報、対応環境を原文で確認します。
  3. 入力と成果物を指定し、出力を資料と照合します。
npx skills add https://github.com/mohitmishra786/low-level-dev-skills --skill cuda-debugging

開始前の確認

  • 導入にはモデル契約、API枠、ファイルへのアクセス許可は含まれません。
  • スター数や導入数は活動の指標であり、独立した品質・安全性試験ではありません。
リポジトリの原文

見出し、例、コード、表、リンク、参照画像を含む原文を表示しています。

CUDA Debugging

Purpose

Guide agents through debugging CUDA programs with cuda-gdb for interactive GPU thread inspection, NVIDIA Compute Sanitizer for automated memory and race detection, GPU core dump analysis, device-side printf, and triaging common CUDA runtime error codes.

When to Use

  • cudaErrorIllegalAddress (700) or segmentation fault on device
  • Intermittent correctness failures in multi-threaded GPU code
  • Debugging race conditions between warps or between host and device
  • Stepping through kernel code line-by-line with cuda-gdb
  • Validating uninitialized memory reads with initcheck
  • Kernel hang or cudaErrorLaunchTimeout (702)

Workflow

1. Build for debugging

bash
# Debug build — disables optimizations, enables device debug
nvcc -G -g -O0 -arch=sm_80 -o app_debug main.cu

# Sanitizer-friendly build (lineinfo helps reports)
nvcc -lineinfo -g -O2 -arch=sm_80 -o app_san main.cu

-G is required for cuda-gdb source-level stepping. Sanitizers work with optimized builds but -G gives clearer line numbers.

2. Compute Sanitizer — automated checks

bash
# Memory errors (OOB, misaligned, leak)
compute-sanitizer --tool memcheck ./app_san

# Shared memory and global memory races
compute-sanitizer --tool racecheck ./app_san

# Uninitialized memory reads
compute-sanitizer --tool initcheck ./app_san

# Synchronization errors (missing __syncthreads)
compute-sanitizer --tool synccheck ./app_san

# Verbose with source correlation
compute-sanitizer --tool memcheck --show-reachable=yes --log-file san.log ./app_san

Typical memcheck output:

======== Invalid __global__ write of size 4
========     at 0x1a0 in vector_add(vector_add.cu:12)
========     by thread (0,0,0) in block (0,0,0)
======== Address 0x7f... is out of bounds

3. cuda-gdb interactive session

bash
# Launch under cuda-gdb
cuda-gdb ./app_debug

# Or attach to running process
cuda-gdb -p <pid>

Essential commands:

gdb
# Break at kernel entry
(cuda-gdb) break vector_add
(cuda-gdb) run

# Focus on GPU threads
(cuda-gdb) info cuda kernels
(cuda-gdb) cuda kernel 0
(cuda-gdb) cuda thread (0,0,0)   # block (x,y,z), thread (x,y,z)

# Inspect device memory
(cuda-gdb) print data[i]
(cuda-gdb) x/10f d_ptr

# Step in kernel
(cuda-gdb) cuda step
(cuda-gdb) cuda next

# All threads in block
(cuda-gdb) info cuda threads
(cuda-gdb) cuda thread (0,0,5)

4. Device printf

c
__global__ void debug_kernel(float *data, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) {
        if (i < 5)  // limit output
            printf("thread %d: data[%d] = %f\n", i, i, data[i]);
        data[i] *= 2.0f;
    }
}
bash
# Buffer size for printf (default may truncate)
cuda-gdb) set cuda printf_buffer_size 16777216

Flush with cudaDeviceSynchronize() before checking output. Excessive printf from all threads will overwhelm the buffer.

5. Error code triage

CodeNameCommon cause
700cudaErrorIllegalAddressOOB access, use-after-free, bad pointer
701cudaErrorLaunchOutOfResourcesToo much shared mem or registers per block
702cudaErrorLaunchTimeoutInfinite loop, TDR watchdog (Windows/default Linux)
719cudaErrorLaunchFailureAssert in kernel, stack overflow
c
// Always check after launch
kernel<<<grid, block>>>(args);
cudaError_t err = cudaGetLastError();
if (err != cudaSuccess)
    fprintf(stderr, "launch: %s\n", cudaGetErrorString(err));
cudaDeviceSynchronize();
err = cudaGetLastError();
if (err != cudaSuccess)
    fprintf(stderr, "exec: %s\n", cudaGetErrorString(err));

6. GPU core dumps

bash
# Enable coredump (driver 450+)
export CUDA_ENABLE_COREDUMP_ON_EXCEPTION=1
export CUDA_COREDUMP_FILE=/tmp/cuda_coredump_%h.%p

./app_san   # crash generates dump

# Analyze with cuda-gdb
cuda-gdb ./app_san /tmp/cuda_coredump_hostname.pid
(cuda-gdb) cuda coredump load /tmp/cuda_coredump_hostname.pid
(cuda-gdb) bt
(cuda-gdb) info cuda kernels

7. Debugging decision tree

Crash or wrong results?
├── Consistent wrong values → logic bug; use printf or cuda-gdb
├── Intermittent / depends on size → OOB or race
│   ├── compute-sanitizer --tool memcheck
│   └── compute-sanitizer --tool racecheck
├── Hang / timeout 702 → infinite loop or barrier mismatch
│   └── synccheck; audit __syncthreads paths
└── Works in debug (-G), fails in release → uninitialized mem or race
    └── initcheck + racecheck on release build

8. Multi-GPU and MIG notes

bash
# Isolate GPU
CUDA_VISIBLE_DEVICES=0 compute-sanitizer --tool memcheck ./app

# MIG instances appear as separate devices
nvidia-smi -L

Common Problems

SymptomCauseFix
cuda-gdb can't break in kernelBuilt without -GRebuild with nvcc -G -g -O0
Sanitizer reports no errors but crash persistsAsync error delayedAdd cudaDeviceSynchronize() after kernel
printf shows nothingBuffer full or no syncLimit prints; increase buffer; sync
racecheck false positive on atomicsNon-atomic RMWUse atomicAdd/atomicCAS
Attach failsProcess not in CUDA contextBreak after first cudaMalloc
TDR timeout on WindowsLong-running kernelSplit kernel; cudaDeviceSetLimit or regedit TDR

Related Skills

  • skills/gpu/cuda — kernel patterns, memory hierarchy, launch config
  • skills/gpu/cuda-profiling — performance after correctness is verified
  • skills/gpu/gpu-memory-model — understanding races and coalescing
  • skills/debuggers/gdb — host-side GDB commands shared with cuda-gdb
  • skills/runtimes/sanitizers — ASan/TSan concepts for host code
  • skills/kernel/kernel-debugging — kgdb/kprobes for driver-level issues

タスクに関連する項目

同じリポジトリから

関連する Skills

すべての Skills
mohitmishra786
コミュニティ

low-power-embedded

Low-power embedded skill for sleep modes and energy optimization. Use when configuring MCU sleep/stop/standby, peripheral clock gating, wake-up sources, or measuring firmware current draw. Activates on queries about WFI, STOP mode, STM32 PWR, nRF sleep, wake-up EXTI, or reducing embedded power consumption.

1時間の導入数
1
GitHub Stars
296
更新日
11月20日
mohitmishra786
コミュニティ

core-dumps

Core dump analysis skill for production crash triage. Use when loading core files in GDB or LLDB, enabling core dump generation on Linux/macOS, mapping symbols with debuginfo or debuginfod, or extracting backtraces from crashes without re-running the program. Activates on queries about core files, ulimit, coredumpctl, debuginfod, crash triage, or analyzing segfaults from production binaries.

1時間の導入数
2
GitHub Stars
267
更新日
6月27日
mohitmishra786
コミュニティ

custom-allocators

Custom allocator skill for memory allocation strategies. Use when implementing pool/slab/arena allocators, tuning jemalloc/mimalloc, writing Rust GlobalAlloc, or benchmarking allocator performance. Activates on queries about jemalloc, mimalloc, tcmalloc, arena allocator, GlobalAlloc, or fragmentation.

1時間の導入数
2
GitHub Stars
267
更新日
6月27日
mohitmishra786
コミュニティ

ebpf

eBPF skill for Linux observability and networking. Use when writing eBPF programs with libbpf or bpftrace, attaching kprobes/tracepoints/XDP hooks, debugging verifier errors, working with eBPF maps, or achieving CO-RE portability across kernel versions. Activates on queries about eBPF, bpftool, bpftrace, XDP programs, libbpf, verifier errors, eBPF maps, or kernel tracing with BPF.

1時間の導入数
2
GitHub Stars
267
更新日
6月27日