posthog/ai-plugin

investigating-metric-anomalies

Investigates server/infrastructure metric anomalies in PostHog Metrics — from "this metric is rising/dropping/spiking" or a fired alert to a probable cause with evidence.

소스 보기

사용 방법

  1. 설치 명령을 복사하고 자신의 에이전트 환경에 설치하기 전에 소스 패키지를 확인하세요.
  2. 필요한 도구, 인증 정보, 지원 환경은 원본 문서에서 확인하세요.
  3. 입력과 원하는 산출물을 설명하고 출처와 결과를 대조하세요.
npx skills add https://github.com/posthog/ai-plugin --skill investigating-metric-anomalies

시작 전 확인

  • 설치에는 모델 구독, API 한도, 파일 접근 권한이 포함되지 않습니다.
  • 별점 수와 설치 수는 활동 지표이며 독립적인 품질·보안 검증이 아닙니다.
원본 Skill 문서

원본 저장소의 제목, 예시, 코드, 표, 링크, 이미지를 유지해 표시합니다.

Investigating metric anomalies

The job: go from a metric symptom ("ingestion lag is rising") to a probable cause with evidence, fast. The metric tells you what and when; logs and traces tell you why. Follow the loop below — it front-loads the cheap, high-information calls and only fans out when the blast radius is unclear.

The loop

1. Pin down the metric

If you have the exact metric name, skip ahead. Otherwise call metric-names-list with a substring from the symptom (lag, error, latency, queue). The returned metric_type decides the lens: counters (sum) are only meaningful as rate/increase, gauges as avg, histograms as histogram_quantile.

2. Characterize first — one call, three answers

Call characterize-metric-anomaly with the metric name and anomalyFrom (the alert fire time, or when the user says it started looking wrong; subtract some margin if unsure). It compares against the preceding window by default and answers:

  • How bad: direction, change_ratio, anomaly_peak vs baseline_mean. If direction is flat, your window or metric is wrong — widen the window, or compare against the same window yesterday via baselineFrom/baselineTo (daily-pattern metrics often look "anomalous" against the immediately-preceding hours).
  • When: onset_time — treat this timestamp as the pivot for everything that follows.
  • Where: top_movers — label values whose behavior changed. One mover (a single pod, shard, or endpoint) means a localized culprit; everything moving together means a shared cause (an upstream dependency, a deploy, infra).

3. Sharpen with targeted metric queries

Use query-metrics to test the hypotheses the report raises:

  • Drill a mover: re-query with filters pinning the suspicious label value, grouped by a second key, to localize further (pod → container, endpoint → status code).
  • Normalize: a rising error count means nothing if traffic doubled — use clauses + formula (errors / requests) to separate rate changes from volume changes.
  • Check the neighbors: query the obvious companion metrics over the same window (for lag: throughput and error counters of the same service; for latency: request rate and saturation gauges). Use the same interval so the grids align visually.

4. Correlate across signals at the onset

Pivot into logs and traces using the same service and a window bracketing `onset_time` (a few buckets before, through the peak):

  • Logs: use query-logs (follow its own discover-first workflow) filtered to the implicated service.name and window, severity error first, then warn. Restarts, crash loops, connection errors, and deploy markers right before onset are the classic causes. Widen to other services in the request path if the service's own logs are clean.
  • Traces: use the APM span tools (query-apm-spans etc.) for the same service/window — slow or erroring spans show which dependency degraded, and a trace_id from an exemplar or log line links a concrete request across all three signals.

5. Conclude with evidence, not vibes

State: the symptom (metric, magnitude, onset), the probable cause (what you found in logs/traces and how its timing aligns with the onset), the blast radius (which services/labels are affected, from the movers and grouped queries), and the confidence level. If the cause is still ambiguous, say which hypothesis the evidence favors and what would disambiguate (e.g. "the lag began draining at 20:12 — consistent with a consumer restart; check who restarted it").

Worked example: "ingestion lag is rising — why?"

  1. metric-names-list with value: "lag" → logs_rate_limiter_message_lag_seconds (histogram) and friends.
  2. characterize-metric-anomaly on it with anomalyFrom = alert time → direction up, change ratio 40x, onset_time 20:10, top mover service_name = logs-ingestion (the other services' lag stayed flat) — so the logs consumer specifically is behind, not the whole pipeline.
  3. query-metrics: rate of the consumer's throughput counter over the same window → throughput was zero during the gap and spiked after onset: the consumer wasn't slow, it was down, and the "rising lag" is it draining the backlog.
  4. query-logs for service.name = logs-ingestion (and its neighbors) around 20:00–20:15 → process exit + restart lines at the gap boundaries.
  5. Conclusion: consumer outage 20:01–20:10 (restart visible in logs); lag spike is backlog drain, self-recovering; affected signal: logs freshness only — no data loss (topic retained messages). Evidence: zero throughput during the window, message-age spike equal to outage duration, restart log lines.

Pitfalls

  • Counters reset on process restart — rate/increase already handle this; never eyeball raw cumulative counter values.
  • A metric that stops reporting is not "zero" — a gap in series with top_movers showing a vanished label value means the emitter died; pivot to logs immediately.
  • Don't trust a single aggregation: a flat avg can hide a screaming p95. For latency-like gauges and histograms, characterize the tail too.
  • Scraped Prometheus metrics arrive ~15–60s behind real time; don't read the last bucket of a series as "current".

관련 작업

같은 저장소의 Skills

더 많은 Skills

모든 Skills
posthog
공식

exploring-llm-costs

Investigate LLM spend in PostHog — total cost over time, cost by model, provider, user, trace, or custom dimension, token and cache-hit economics, and cost regressions. Use when the user asks "how much are we spending on LLMs?", "which model / user / feature is most expensive?", "why did cost spike?", wants to build a cost dashboard or alert, or pastes a trace URL and asks about its cost.

1시간 설치 수
2
GitHub Stars
98
업데이트
11월 6일
posthog
공식

exploring-llm-traces

Debug and inspect LLM/AI agent traces using PostHog's MCP tools. Use when the user pastes a trace or session URL (e.g. /ai-observability/traces/ or /ai-observability/sessions/ ), asks to debug a trace, figure out what went wrong, check if an agent used a tool correctly, verify context/files were surfaced, inspect subagent behavior, investigate LLM decisions, or analyze token usage and costs. Also use when raw SQL/HogQL against events.properties.$aiinput / $aioutputchoices returns empty — message

1시간 설치 수
2
GitHub Stars
98
업데이트
11월 6일
posthog
공식

instrument-integration

Add PostHog SDK integration to your application. Use when setting up PostHog for the first time or reviewing PRs that need PostHog initialization. Covers SDK installation, provider setup, and basic configuration for any framework.

1시간 설치 수
2
GitHub Stars
98
업데이트
11월 6일
posthog
공식

investigating-error-issue

Investigates a single PostHog error tracking issue end-to-end. Use when the user provides an issue ID or pastes an issue URL (/error-tracking/ ) and wants to understand the error — who it affects, what triggers it, when it started, whether it correlates with a release, browser, OS, or feature flag, and what the next step should be. Pulls aggregated metrics, sample exception events, segment breakdowns, linked replays, and synthesizes a hypothesis-grade summary in one pass.

1시간 설치 수
2
GitHub Stars
98
업데이트
11월 6일