posthog/ai-plugin

exploring-llm-traces

Debug and inspect LLM/AI agent traces using PostHog's MCP tools.

查看源码

使用方法

  1. 复制安装命令,在自己的智能体环境安装前先检查源码包。
  2. 从原始文档核对所需工具、凭据和支持的宿主。
  3. 说明输入材料与期望交付物,并对照来源检查输出。
npx skills add https://github.com/posthog/ai-plugin --skill exploring-llm-traces

开始前需了解

  • 安装不包含模型订阅、API 额度或访问文件的权限。
  • 仓库星数和安装量反映活跃情况,不等于独立质量或安全测试。
仓库原始内容

按源仓库内容呈现,保留标题、案例、代码、表格、链接以及原文引用的演示图片。

Exploring LLM traces with MCP tools

PostHog captures LLM/AI agent activity as traces. Each trace is a tree of events representing a single AI interaction — from the top-level agent invocation down to individual LLM API calls.

Available tools

ToolPurpose
posthog:query-llm-traces-listSearch and list traces; can return large multi-trace payloads
posthog:query-llm-traceGet a single trace by ID with full event tree
posthog:read-data-schemaDiscover custom event/person properties before filtering
posthog:execute-sqlAd-hoc SQL for complex trace analysis

Event hierarchy

See the event reference for the full schema.

text
$ai_trace (top-level container)
  └── $ai_span (logical groupings, e.g. "RAG retrieval", "tool execution")
        ├── $ai_generation (individual LLM API call)
        └── $ai_embedding (embedding creation)

Events are linked via $ai_parent_id → parent's $ai_span_id or $ai_trace_id.

Workflow: debug a trace or session from a URL

Step 1 — Classify the URL

First inspect the path. Do not treat every UUID-looking value as a trace ID.

  • /ai-observability/traces/<trace_id> or legacy /llm-analytics/traces/<trace_id> / /llm-observability/traces/<trace_id> is a single trace. Fetch it with posthog:query-llm-trace.
  • /ai-observability/sessions/<session_id> or legacy /llm-analytics/sessions/<session_id> is an AI session, not a trace. Fetch traces with posthog:query-llm-traces-list filtered by event property $ai_session_id.

Preserve date_from / date_to query parameters from the URL when present. If none are present but the URL has a timestamp query parameter, use that timestamp as the anchor and query an absolute window around it, for example timestamp - 36h to timestamp + 36h. This handles exact session links whose UI timestamp may be offset from the stored event timestamps while keeping the query bounded. If the URL has neither explicit dates nor timestamp, use a safe default like {"date_from": "-7d"}.

For exact trace and session URLs, skip schema discovery for the standard $ai_* fields used below. These are AI observability built-ins, not project-specific custom properties.

Step 2 — Browse trace summaries

Explicitly set detail: "summary" when browsing traces. This returns event metadata and leaves the conversation content out, so browsing does not spend context on prompts and outputs. Omitting detail still returns full detail for compatibility with existing callers.

For a trace URL, call posthog:query-llm-trace with:

json
{
  "traceId": "<trace_id>",
  "detail": "summary",
  "dateRange": { "date_from": "-7d" }
}

For a session URL, call posthog:query-llm-traces-list with:

json
{
  "detail": "summary",
  "dateRange": { "date_from": "<timestamp_minus_36h>", "date_to": "<timestamp_plus_36h>" },
  "filterTestAccounts": false,
  "limit": 20,
  "properties": [{ "type": "event", "key": "$ai_session_id", "value": ["<session_id>"], "operator": "exact" }]
}

Use the URL's date_from / date_to values in the session query if present. If the URL only has timestamp, calculate the absolute date range from that timestamp instead of using a relative range like -1h. Set filterTestAccounts: false for an exact URL so the requested trace is not hidden by account filters.

The result contains trace and event metadata only. Prompts, outputs, span states, $ai_error and $ai_feedback_text are left out, and their names are listed in _summaryOmittedKeys beside the property bag. A trace with _detail: { "mode": "summary" } carries no conversation content at all. Read the content with detail: "full".

From the result you get:

  • Every event with its type ($ai_span, $ai_generation, etc.)
  • Span names ($ai_span_name) — these are the tool/step names
  • Latency, error flags, models used
  • Parent-child relationships via $ai_parent_id
  • _posthogUrl — always include this in your response so the user can click through to the UI

Step 3 — Read the content needed for the investigation

Once you have selected a trace, request posthog:query-llm-trace with detail: "full" before inspecting exact tool arguments, checking which context the model received, or searching conversation content:

json
{
  "traceId": "<trace_id>",
  "detail": "full",
  "dateRange": { "date_from": "-7d" }
}

Preserve the date range from the original URL or discovery query instead of copying the example range. Keep relevant property filters to narrow the read. If the user already identified the trace and needs exact content, you can request full detail directly.

Both modes enforce response size limits. Check the omission and truncation markers before drawing conclusions. A dropped event, a name listed in _summaryOmittedKeys or _redactedKeys, or a keyword missing from a summary is not evidence that it was absent from the trace. If full detail is still truncated, narrow the query to the relevant events or open _posthogUrl for the complete data. Both detail modes also withhold the properties that are not AI payload, described in Withheld properties below.

Step 4 — Parse large full-detail results with scripts

When the result is persisted to a file (large traces with full $ai_input/$ai_output_choices), use the parsing scripts to explore it.

Start with the summary to get the full picture, then drill into specifics:

bash
# 1. Overview: metadata, tool calls, final output, errors
python3 scripts/print_summary.py /path/to/persisted-file.json

# 2. Timeline: chronological event list with truncated I/O
python3 scripts/print_timeline.py /path/to/persisted-file.json

# 3. Drill into a specific span's full input/output
SPAN="tool_name" python3 scripts/extract_span.py /path/to/persisted-file.json

# 4. Full conversation with thinking blocks and tool calls
python3 scripts/extract_conversation.py /path/to/persisted-file.json

# 5. Search for a keyword across the returned properties
SEARCH="keyword" python3 scripts/search_traces.py /path/to/persisted-file.json

All scripts support MAX_LEN=N env var to control truncation (0 = unlimited).

Withheld properties

The two trace tools return the AI payload only. Only the $ai_* properties PostHog's taxonomy defines reach you, plus $ai_generation_id, $ai_cache_read_cost_usd, $ai_cache_creation_cost_usd, $ai_effort, $session_id, $lib and $lib_version. Every other event property, and every person property, is withheld in both detail modes, and its name is listed in _redactedKeys beside the property bag. $ai_base_url and $ai_request_url arrive without their query string.

This covers a project's own custom properties, such as project_id or conversation_id. A withheld property is unchanged in PostHog. It still works as a filter here, and you can read its value in the PostHog UI or with execute-sql. A name in _redactedKeys means the property is on the event and only its value is withheld. search_traces.py cannot find a keyword that appears only in a withheld property.

Investigation patterns

"Did the agent use the tool correctly?"

  1. Find the $ai_span for the tool call (look at $ai_span_name)
  2. Check $ai_input_state — what arguments were passed to the tool?
  3. Check $ai_output_state — what did the tool return?
  4. Check $ai_is_error — did the tool call fail?

"Was the context correct?" / "Were the right files surfaced?"

  1. Find the $ai_generation event where the LLM made the decision
  2. Check $ai_input — this is the full message history the LLM saw
  3. Look at preceding $ai_span events for retrieval/search steps
  4. Check their $ai_output_state — what content was retrieved and fed to the LLM?

"Did the subagent work?"

  1. In the structural overview, find spans that are children of other spans (via $ai_parent_id)
  2. The parent span is the orchestrator; child spans are subagent steps
  3. Check each child's $ai_output_state and $ai_is_error
  4. If a child span contains $ai_generation events, those are the subagent's LLM calls

"Why did the LLM say X?"

  1. Use search_traces.py to find where the text appears: SEARCH="the text" python3 scripts/search_traces.py FILE
  2. This shows which event and property path contains it
  3. Check the $ai_input of that generation to see what the LLM was told before it said X

Constructing UI links

The trace tools return _posthogUrl — always surface this to the user.

You can also construct links manually:

  • Trace detail: https://app.posthog.com/ai-observability/traces/<trace_id>?timestamp=<url_encoded_timestamp>&event=<optional_event_id>
  • Traces list with filters: returned in _posthogUrl from query-llm-traces-list

The timestamp query param is required — use the createdAt of the earliest event in the trace, URL-encoded (e.g. timestamp=2026-04-01T19%3A39%3A20Z).

When presenting findings, always include the relevant PostHog URL so the user can verify.

Finding traces

Use posthog:query-llm-traces-list with detail: "summary" to search and filter traces.

CRITICAL: Never assume event names, property names, or property values from training data. Every project instruments different custom properties. For open-ended searches and custom filters, call posthog:read-data-schema first to discover what properties and values actually exist in the project's data before constructing filters.

The exception is exact AI observability trace/session URLs: use the built-in $ai_trace_id / $ai_session_id fields directly and skip schema discovery.

Discovering the schema first

Before filtering traces, discover what's available:

  1. Confirm AI events exist — call posthog:read-data-schema with kind: "events" and look for $ai_* events
  2. Find filterable properties — call posthog:read-data-schema with kind: "event_properties" and event_name: "$ai_generation" (or another AI event) to see what properties are captured
  3. Get actual values — call posthog:read-data-schema with kind: "event_property_values", event_name: "$ai_generation", and property_name: "$ai_model" to see real model names in use

Only then construct the query-llm-traces-list call with property filters.

This is especially important for custom properties like project_id, conversation_id, user_tier, etc. — these vary per project and cannot be guessed. A custom property works as a filter even though its value is withheld from the response.

Do not confirm $ai_* properties, but confirm any other like email of a person.

By filters

json
posthog:query-llm-traces-list
{
  "detail": "summary",
  "dateRange": {"date_from": "-1h"},
  "filterTestAccounts": true,
  "limit": 20,
  "properties": [
    {"type": "event", "key": "$ai_model", "value": "gpt-4o", "operator": "exact"}
  ]
}

Multiple filters are AND-ed together:

json
posthog:query-llm-traces-list
{
  "detail": "summary",
  "dateRange": {"date_from": "-1h"},
  "filterTestAccounts": true,
  "properties": [
    {"type": "event", "key": "$ai_provider", "value": "anthropic", "operator": "exact"},
    {"type": "event", "key": "$ai_is_error", "value": ["true"], "operator": "exact"}
  ]
}

You can also filter by person properties (discover them via read-data-schema with kind: "entity_properties" and entity: "person"):

json
posthog:query-llm-traces-list
{
  "detail": "summary",
  "dateRange": {"date_from": "-1h"},
  "filterTestAccounts": true,
  "properties": [
    {"type": "person", "key": "email", "value": "@company.com", "operator": "icontains"}
  ]
}

By external identifiers

Customers often store their own IDs as event or person properties. Use posthog:read-data-schema to discover what custom properties exist, then filter:

  1. Call posthog:read-data-schema with kind: "event_properties" and event_name: "$ai_trace" to find custom properties
  2. Review the returned properties and their sample values
  3. Construct the filter using the discovered property key and a known value
json
posthog:query-llm-traces-list
{
  "detail": "summary",
  "dateRange": {"date_from": "-7d"},
  "properties": [
    {"type": "event", "key": "project_id", "value": "proj_abc123", "operator": "exact"}
  ]
}

For more complex SQL patterns, read these references:

Parsing large trace results

Trace tool results are JSON. When too large to read inline, Claude Code persists them to a file. Use a full-detail response for content extraction and keyword searches. The scripts cannot recover content omitted from a summary, withheld from a property bag, or dropped by truncation.

Persisted file format

json
[{ "type": "text", "text": "{\"results\": [...], \"_posthogUrl\": \"...\"}" }]

Trace JSON structure

text
results (array for list, object for single trace)
  ├── id, traceName, createdAt, totalLatency, totalCost
  ├── inputState, outputState (trace-level state)
  └── events[]
        ├── event ($ai_span | $ai_generation | $ai_embedding | $ai_metric | $ai_feedback)
        ├── id, createdAt
        └── properties
              ├── $ai_span_name, $ai_latency, $ai_is_error
              ├── $ai_input_state, $ai_output_state (span tool I/O)
              ├── $ai_input, $ai_output_choices (generation messages)
              ├── $ai_model, $ai_provider
              └── $ai_input_tokens, $ai_output_tokens, $ai_total_cost_usd

Available scripts

ScriptPurposeUsage
`print_summary.py`Aggregate list/session totals, trace metadata, tool calls, errors, and final LLM outputpython3 scripts/print_summary.py FILE
`print_timeline.py`Chronological event timeline with I/O summariespython3 scripts/print_timeline.py FILE
`extract_span.py`Full input/output of a specific span by nameSPAN="name" python3 scripts/extract_span.py FILE
`extract_conversation.py`LLM messages with thinking blocks and tool callspython3 scripts/extract_conversation.py FILE
`search_traces.py`Find a keyword across all event propertiesSEARCH="keyword" python3 scripts/search_traces.py FILE
`show_structure.py`Show JSON keys and types without values`cat blob.json \python3 scripts/show_structure.py`

Tips

  • Always set dateRange — queries without a time range are slow. Use narrow windows (-30m, -1h) for broad listing queries; wider windows (-7d, -30d) are fine for narrow queries filtered by trace ID or specific property values
  • Always include the _posthogUrl in your response so the user can click through
  • $ai_input_state / $ai_output_state on spans contain tool call inputs and outputs
  • $ai_input / $ai_output_choices on generations contain the full LLM conversation — can be megabytes; when the result is persisted to a file, use the parsing scripts
  • In raw SQL, heavy content ($ai_input / $ai_output / $ai_output_choices / $ai_input_state / $ai_output_state / $ai_tools) lives only on the posthog.ai_events table, not events.properties — see the event reference for the column mapping and trace-id-anchored query patterns
  • Use filterTestAccounts: true to exclude internal/test traffic when searching
  • When an investigation turns up a failure that will happen again, not a one-off, offer to catch it with an online evaluation. Load creating-online-evaluations to set it up, so new traces get scored for it without anyone digging
  • $ai_trace events are NOT in the events array — their data is surfaced via trace-level inputState, outputState, and traceName
来自同一仓库

更多 Skills

全部 Skills
posthog
官方

exploring-llm-costs

Investigate LLM spend in PostHog — total cost over time, cost by model, provider, user, trace, or custom dimension, token and cache-hit economics, and cost regressions. Use when the user asks "how much are we spending on LLMs?", "which model / user / feature is most expensive?", "why did cost spike?", wants to build a cost dashboard or alert, or pastes a trace URL and asks about its cost.

1小时安装量
2
GitHub Stars
98
最近更新
11月6日
posthog
官方

instrument-integration

Add PostHog SDK integration to your application. Use when setting up PostHog for the first time or reviewing PRs that need PostHog initialization. Covers SDK installation, provider setup, and basic configuration for any framework.

1小时安装量
2
GitHub Stars
98
最近更新
11月6日
posthog
官方

investigating-error-issue

Investigates a single PostHog error tracking issue end-to-end. Use when the user provides an issue ID or pastes an issue URL (/error-tracking/ ) and wants to understand the error — who it affects, what triggers it, when it started, whether it correlates with a release, browser, OS, or feature flag, and what the next step should be. Pulls aggregated metrics, sample exception events, segment breakdowns, linked replays, and synthesizes a hypothesis-grade summary in one pass.

1小时安装量
2
GitHub Stars
98
最近更新
11月6日
posthog
官方

investigating-replay

Investigates a session recording by gathering metadata, person profile, same-session events, and linked error tracking issues in one pass. Use when a user provides a recording or session ID and wants to understand what happened — who the user was, what they did, what errors occurred, and whether there are related error tracking issues. Replaces the manual chain of session-recording-get, persons-retrieve, execute-sql, and query-error-tracking-issues-list.

1小时安装量
2
GitHub Stars
98
最近更新
11月6日