posthog/ai-plugin

exploring-llm-costs

Investigate LLM spend in PostHog — total cost over time, cost by model, provider, user, trace, or custom dimension, token and cache-hit economics, and cost regressions.

Quelltext ansehen

Verwendung

  1. Installationsbefehl kopieren und das Quellpaket vor der Installation in der eigenen Agentenumgebung prüfen.
  2. Benötigte Werkzeuge, Zugangsdaten und Hosts im Original prüfen.
  3. Eingabe und gewünschtes Ergebnis beschreiben und die Ausgabe mit den Quellen abgleichen.
npx skills add https://github.com/posthog/ai-plugin --skill exploring-llm-costs

Vor dem Start

  • Die Installation umfasst weder Modellabo noch API-Kontingent oder Dateizugriffsrechte.
  • Sterne und Installationen zeigen Aktivität, keine unabhängige Qualitäts- oder Sicherheitsprüfung.
Originales Skill-Dokument

Aus dem Quell-Repository gerendert; Überschriften, Beispiele, Code, Tabellen, Links und Bilder bleiben erhalten.

Exploring LLM costs

PostHog attaches per-call cost metadata to every $ai_generation and $ai_embedding event at ingestion time. Every cost question reduces to an aggregation over those two event types — the interesting variation is only in how you group, filter, and compare.

This skill covers the common cost investigations: total spend, breakdowns (model, provider, user, trace, custom property), token and cache-hit analysis, regression debugging, and materializing results as insights, dashboards, or alerts.

Tools

ToolPurpose
posthog:query-trendsNative cost trends with supported sums, breakdowns, and comparisons
posthog:execute-sqlAd-hoc HogQL for any cost aggregation — the workhorse of this skill
posthog:query-llm-traces-listList traces with rolled-up cost, token, and error metrics
posthog:query-llm-traceCost breakdown of a single trace across all its events
posthog:read-data-schemaDiscover which custom properties exist for breakdowns
posthog:insight-createMaterialize a cost chart as a saved insight
posthog:dashboard-createBundle cost insights into a dashboard
posthog:alert-createAlert when cost crosses a threshold
posthog:generate-app-urlBuild region- and project-qualified links back to the UI

Core rules

Three rules cover most of what goes wrong:

  • Sum `$ai_total_cost_usd` for rollups, never the components. Components drop

request and web-search fees. The UI's cost cells sum $ai_total_cost_usd over event IN ('$ai_generation', '$ai_embedding'); mirror that. Full schema and rationale in cost properties.

  • Always include both `$ai_generation` and `$ai_embedding` in cost queries

unless the project demonstrably does not use embeddings — missing them silently under-counts. $ai_trace and $ai_span carry no rollup cost; some SDK wrappers duplicate $ai_total_cost_usd onto $ai_trace so don't include it in rollups or you'll double-count.

  • Always set a time range. Cost queries without one scan the full events table.

$ai_total_cost_usd is set at ingestion via one of three paths (passthrough, custom pricing, automatic lookup). When a cost looks wrong, read $ai_cost_model_source first — see cost sources for the precedence rules and a diagnostic query.

Cache-hit math depends on whether the provider reports cache tokens inclusively or exclusively of $ai_input_tokens. Always branch on the per-event $ai_cache_reporting_exclusive flag, never on provider name — see cache accounting for the exclusive-vs-inclusive formula.

distinct_id is the canonical user dimension. Customers often attach custom properties (feature, tenant_id, workflow_name) — discover them with posthog:read-data-schema before grouping. Don't guess names.

Workflow: total spend in a window

sql
posthog:execute-sql
SELECT round(sum(toFloat(properties.$ai_total_cost_usd)), 4) AS total_cost_usd
FROM events
WHERE event IN ('$ai_generation', '$ai_embedding')
    AND timestamp >= now() - INTERVAL 30 DAY

Workflow: cost breakdowns

Every cost question is a variation of the same template — group by a dimension, aggregate $ai_total_cost_usd. See breakdown patterns for ready-to-run recipes:

  • Cost over time (daily)
  • Cost by model
  • Cost by user (top spenders)
  • Cost by trace (top expensive traces)
  • Cost by custom dimension
  • Cost-per-call distribution
  • Input vs output vs cache economics

Workflow: inspect a single trace's cost

When the user pastes a trace URL and asks about its cost, fetch the trace and surface the per-event breakdown:

json
posthog:query-llm-trace
{ "traceId": "<trace_id>", "dateRange": {"date_from": "-30d"} }

Sum $ai_total_cost_usd across the returned events, grouped by span name or model, to show which step(s) drove the cost. The trace response already includes totalCost as a convenience.

Workflow: debug a cost regression

"Our LLM bill jumped — why?" is almost always one of: more calls, bigger prompts, a new model, or a change in cache-hit rate. Work through them in order — see regression debugging for the 5-step playbook.

Workflow: materialize as an insight, dashboard, or alert

After ad-hoc queries answer the question, persist them as insights, bundle into a dashboard, or wire up alerts. See materializing for ready-to-run JSON for posthog:insight-create, posthog:dashboard-create, and posthog:alert-create.

Constructing UI links

Never hand-write https://app.posthog.com/... links. That host drops the region and the project prefix, so the user is redirected to login instead of the page you meant.

  • Prefer the canonical URL the tool returns. query-llm-traces-list and query-llm-trace

return _posthogUrl — surface that value. For a single trace, append ?timestamp=<url_encoded_iso> (the trace's earliest event time) to that URL; the returned link carries no timestamp, and without one the trace page scans from a fixed early date instead of the ten-minute window around the trace.

  • Otherwise build the link with `generate-app-url`. It resolves the correct region host and

/project/<id>/ prefix (e.g. https://us.posthog.com/project/2/ai-observability/traces). Pass concrete ids via params, never inline them into the path.

  • Dashboard: generate-app-url {url: "/ai-observability/dashboard"}
  • Traces list (sort by cost): generate-app-url {url: "/ai-observability/traces"}
  • Generations list: generate-app-url {url: "/ai-observability/generations"}
  • Users list (per-user cost): generate-app-url {url: "/ai-observability/users"}
  • Single trace: generate-app-url {url: "/ai-observability/traces/{id}", params: {id: "<trace_id>"}}

generate-app-url cannot express query params, so append the ?timestamp=<url_encoded_iso> described above to a single-trace link yourself.

Always surface a UI link so the user can verify visually.

Keeping this skill current

Provider reporting behavior (which tokens are inclusive vs exclusive, which costs show up where) shifts over time and can differ between SDK versions for the same provider. To avoid rot:

  • Branch on event-level flags ($ai_cache_reporting_exclusive,

$ai_cost_model_source) rather than hardcoded provider or model names. When set, those flags are ingestion's resolved answer for the specific event. $ai_cache_reporting_exclusive is often missing; see cache accounting for the fallback.

  • $ai_total_cost_usd is always authoritative for rollups — prefer it

over summing components, which can drift as new cost categories are added.

  • For anything not covered here (new cost categories, changes to

pricing lookup, provider additions), run posthog:docs-search for "calculating costs" or "AI observability" first rather than trusting a hardcoded rule in this file.

  • If you find this skill contradicting the UI, trust the UI and flag

the skill for an update.

Tips

  • Always set a time range — cost queries without one scan the full events table
  • Token, cost, model, and $ai_trace_id properties are on events — but message content ($ai_input / $ai_output_choices) lives only on the posthog.ai_events table; load the exploring-llm-traces skill if you need content alongside cost
  • Always include $ai_embedding alongside $ai_generation when summing cost; embeddings are cheap per-call but add up at scale
  • Costs are written at ingestion (see Calculating LLM costs) — if $ai_total_cost_usd is missing or zero, read $ai_cost_model_source first: passthrough means the SDK supplied costs; custom means custom token prices; openrouter / manual mean automatic lookup; missing means the model wasn't matched (unusual custom model, fine-tune). Grep: countIf(properties.$ai_total_cost_usd IS NULL) per (model, source)
  • Custom pricing uses per-token prices, not per-million — if a custom-priced model looks ~1M× too expensive or too cheap, that's almost always the bug
  • Exclude errored calls from cost totals only when explicitly asked — providers still charge for many error modes, and including them gives the truthful bill
  • For per-user totals, exclude rows where distinct_id = properties.$ai_trace_id — some SDKs default distinct_id to the trace ID when no user is set
  • Cost is additive across $ai_generation + $ai_embedding events within a trace; summing on $ai_span gives zero. $ai_trace may carry $ai_total_cost_usd from some SDK wrappers — don't include it in rollups or you'll double-count. $ai_evaluation events also carry cost but are not part of the stock UI rollups; include them only when the user explicitly wants evaluation spend in the total
  • Cache-hit rate depends on $ai_cache_reporting_exclusive — resolve it per event, never per group. Most events with a precomputed cost carry no flag, so never default a missing flag to inclusive; use the fallback in cache accounting
  • When answering "why is X expensive?", show the cost and the token split — the user almost always wants to know whether to shrink prompts, shrink outputs, or switch models
  • Before building a custom dashboard, check whether the stock /ai-observability/dashboard tiles already answer the question — re-creating them is churn
  • For large tenants, materialize common cost queries as insights and reuse via insight-query; ad-hoc SQL is fine for one-offs but re-running it on every dashboard load is expensive

References

Related skills

  • `analyzing-expensive-users` — who drives the spend, and whether their usage pattern explains it
  • `exploring-llm-traces` — inspect the expensive traces the breakdowns point at
aus demselben Repository

Weitere Skills

Alle Skills
posthog
Offiziell

exploring-llm-traces

Debug and inspect LLM/AI agent traces using PostHog's MCP tools. Use when the user pastes a trace or session URL (e.g. /ai-observability/traces/ or /ai-observability/sessions/ ), asks to debug a trace, figure out what went wrong, check if an agent used a tool correctly, verify context/files were surfaced, inspect subagent behavior, investigate LLM decisions, or analyze token usage and costs. Also use when raw SQL/HogQL against events.properties.$aiinput / $aioutputchoices returns empty — message

Installationen / 1h
2
GitHub Stars
98
Aktualisiert
6. Nov.
posthog
Offiziell

instrument-integration

Add PostHog SDK integration to your application. Use when setting up PostHog for the first time or reviewing PRs that need PostHog initialization. Covers SDK installation, provider setup, and basic configuration for any framework.

Installationen / 1h
2
GitHub Stars
98
Aktualisiert
6. Nov.
posthog
Offiziell

investigating-error-issue

Investigates a single PostHog error tracking issue end-to-end. Use when the user provides an issue ID or pastes an issue URL (/error-tracking/ ) and wants to understand the error — who it affects, what triggers it, when it started, whether it correlates with a release, browser, OS, or feature flag, and what the next step should be. Pulls aggregated metrics, sample exception events, segment breakdowns, linked replays, and synthesizes a hypothesis-grade summary in one pass.

Installationen / 1h
2
GitHub Stars
98
Aktualisiert
6. Nov.
posthog
Offiziell

investigating-replay

Investigates a session recording by gathering metadata, person profile, same-session events, and linked error tracking issues in one pass. Use when a user provides a recording or session ID and wants to understand what happened — who the user was, what they did, what errors occurred, and whether there are related error tracking issues. Replaces the manual chain of session-recording-get, persons-retrieve, execute-sql, and query-error-tracking-issues-list.

Installationen / 1h
2
GitHub Stars
98
Aktualisiert
6. Nov.