dpm-finder

dpm-finder

Grafana PS tool ranking Prometheus metrics by DPM with per-series breakdown. Source: https://github.com/grafana-ps/dpm-finder

Prerequisites

• Python 3.9+
• A Grafana Cloud Prometheus endpoint URL + numeric stack ID + API key (glc_…, metrics:read scope)

Common Workflows

One-shot analysis (most common)

# 1. Clone + venv + install
git clone https://github.com/grafana-ps/dpm-finder.git
cd dpm-finder
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt

# 2. Configure creds — copy .env_example → .env and fill in:
#    PROMETHEUS_ENDPOINT  https://prometheus-<cluster_slug>.grafana.net  (NOTHING after .net)
#    PROMETHEUS_USERNAME  <numeric stack id>
#    PROMETHEUS_API_KEY   glc_…

# 3. Verify creds before scanning — should return >0 series count
curl -s -u "$PROMETHEUS_USERNAME:$PROMETHEUS_API_KEY" \
  "$PROMETHEUS_ENDPOINT/api/v1/label/__name__/values" | jq '.data | length'

# 4. Run the scan (10-min lookback, 2.0 DPM minimum, top output)
./dpm-finder.py -f json -m 2.0 -t 8 --timeout 120 -l 10

# 5. Read the result — top 10 metrics by DPM
jq -r '.metrics | sort_by(-.dpm) | .[:10][] | "\(.dpm)\t\(.series_count)\t\(.metric_name)"' metric_rates.json

If step 5 is empty, lower -m or confirm the endpoint URL has no trailing path after .net.

Discover stack details with gcx

If gcx is installed it can derive the endpoint + username:

gcx config check          # active stack context
gcx config list-contexts  # all configured stacks
gcx config view           # full config with endpoints

The Prometheus endpoint pattern is https://prometheus-{cluster_slug}.grafana.net. Username is the numeric stack ID.

Without gcx: look up in the Grafana Cloud portal, or query grafanacloud_instance_info{name=~"STACK_NAME.*"} on the usage datasource.

Multi-stack runs

Limit to max 3 concurrent runs to avoid GCloud rate limits. Batch the stacks and wait for each batch before the next.

Interpreting results

• DPM = max data points per minute across that metric's series
• series_count = active time-series count for that metric
• series_detail[] (JSON / text only) = per-label-combination DPM breakdown — use this to spot the offending label
• Sort by DPM descending → noisiest metrics; combine with --cost-per-1000-series to prioritize by spend

Troubleshooting

• 401 / 403 — API key invalid or missing metrics:read; confirm PROMETHEUS_USERNAME is the numeric stack ID
• Timeouts — bump --timeout to 120+ for stacks with thousands of metrics
• HTTP 422 — metric has aggregation rules; tool warns + skips automatically
• Empty results — lower -m; verify endpoint has no trailing path
• Connection errors — exponential backoff retries up to 10 times; persistent failure usually = network/firewall

References

• `references/cli.md` — full flag reference, output-format details, exporter mode, Docker invocation, auto-exclusion rules, retry behavior

Resources

• dpm-finder GitHub
• gcx