python-profiling

Python Profiling Skill

Use this skill when investigating CPU time, wall-clock latency, Python allocations, process memory, object growth, or interpreter-level events. Choose the narrowest tool that can answer the question instead of running every profiler by default.

Repository setup and baseline

The examples target the Python version in python/.python-version and use the virtual environment described in python/README.md:

cd python
pyenv install "$(cat .python-version)"  # only if that version is not installed
pyenv local
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Run an unprofiled baseline before changing code:

python program.py
time python program.py

program.py fetches a Wikipedia page over HTTPS before parsing it. Treat a non-empty top-ten word list as the baseline preflight; [] means the intended parsing workload did not run and the resulting profile is not useful. The current target calls requests.get(...) without a descriptive User-Agent or raise_for_status(), so Wikimedia may return its small HTTP 403 robot-policy response and the program may silently print []. If that happens, stop before profiling and make the workload valid: preferably use a checked-in/local fixture, or update the request to use a policy-compliant descriptive User-Agent and fail on non-success responses. Do not compare a rejection-response profile with a successful-page profile.

Even with a successful response, network, DNS, TLS, server, and response-size variance can dominate timings. For repeatable comparisons, keep the input and environment fixed; never treat one network run as a reliable regression measurement.

The repository's comparison runner is:

./timing.sh

Profiling changes both runtime and memory behavior, so compare profiles only under the same Python build, dependency versions, input, warm-up state, and command-line options.

It invokes cProfile, tracemalloc, Memray, pyinstrument, yappi, Filprofiler, psutil, and guppy3.

Choose by question

For detailed commands, use the focused skills python-cpu-profiling, python-memory-profiling, and python-dtrace-profiling in .claude/skills/.

Reproducible profiling workflow

1. Define the observable: elapsed time, CPU time, peak RSS, Python allocation delta, native allocation, object count, or call sequence.
2. Validate that the unprofiled workload completed its intended work. For this repository, [] is a failed preflight, not a valid fast baseline.
3. Record the baseline command, Python version, dependency lock/requirements, input, host, and profiler options.
4. Run the smallest profiler that measures that observable. Keep output separate from source and name artifacts with the run/configuration.
5. Inspect the profiler's own output format rather than inferring from process RSS alone. Distinguish inclusive/cumulative time from self/tottime and live memory from total allocated memory.
6. Repeat enough times to see variance. For network-bound examples, use a local fixture before comparing code changes.
7. Verify a proposed optimization with the same unprofiled command and at least one profile that explains the change. A faster profile can merely reflect a different input or warmed cache.
8. Remove generated artifacts or keep them in the existing ignored patterns (*.out, memray-*.html, and memray-*.bin under python/). Do not commit credentials, response bodies, or production memory dumps.

Common interpretation traps

• CPU profilers report execution attribution, not automatically user-visible latency. Blocking network, disk, subprocess, and scheduler time may need wall-clock sampling or explicit timing marks.
• cProfile cumulative time includes descendants; tottime is time in the function body. Sort by both when deciding whether a caller or leaf is the useful target.
• A tracemalloc diff measures Python allocator traces between snapshots. It is not equivalent to RSS, and a negative diff means traced allocations were released, not that the process returned all pages to the OS.
• Memray, Filprofiler, and psutil answer different memory questions: allocation provenance, peak behavior, and process counters respectively. Correlate them instead of substituting one for another.
• Sampling output is statistical. Increase duration or repeat workload before concluding that a small frame is absent or faster.
• Profiling a script that performs a live HTTP request measures the remote service and transport as well as local parsing. Keep those costs explicit in reports.

Safety and operational boundaries

Run profilers against a representative but bounded workload first. Avoid attaching py-spy, DTrace, or similar tools to a process without authorization; they may require elevated permissions and can expose arguments, paths, or application data. Use sudo only for the DTrace command that needs it, never for the application itself. Do not run heap/object inspection on sensitive production processes unless the data-handling and retention policy permits it.