按源仓库内容呈现，保留标题、案例、代码、表格、链接以及原文引用的演示图片。
Exploring MCP tool quality
Any MCP server instrumented with PostHog's MCP analytics SDK emits a $mcp_tool_call event on the shared events table every time an agent invokes a tool. There is no dedicated ClickHouse table — every field lives as a $mcp_* property on events, and every tool-quality metric (error rate, latency percentiles, reach) is an aggregation over this one event. This is the data behind the MCP analytics dashboard and tool-quality screens.
For a single tool, prefer the typed tools — posthog:query-mcp-tool-stats (calls, errors, p50/p95, users, sessions, intents), posthog:query-mcp-tool-failures (top error messages by harness), and posthog:query-mcp-tool-daily-stats (day-by-day trend). Each takes a toolName + dateRange, runs the same query runner as the tool-detail UI, and is gated behind the mcp-analytics flag — no hand-written SQL needed.
HogQL via `posthog:execute-sql` is the path for cross-tool questions — the "which tool errors most" ranking below has no typed tool, so rank with SQL, then drill into the worst tool with posthog:query-mcp-tool-stats and posthog:query-mcp-tool-failures. The full property schema and the canonical query recipes live in the shared MCP data reference: `products/posthog_ai/skills/querying-posthog-data/references/models-mcp.md`. That reference is the single source of truth for the $mcp_* schema and the effective-tool-name idiom used below — this skill inlines only the headline "which tool errors most" query for convenience; pull the matrix, latency, and harness recipes from the reference rather than re-deriving them. Read it before writing queries.
The two rules that matter most
- Always use the effective tool name. New-SDK events wrap the real tool in
a single-exec call, so grouping on raw $mcp_tool_name collapses everything under the wrapper. Use:
coalesce(nullIf(toString(properties.$mcp_exec_tool_call_name), ''), toString(properties.$mcp_tool_name))- Always read `$mcp_is_error` via `toBool(...)` and cast
$mcp_duration_ms via toFloat(...). The properties are strings.
Always set a time range — these queries scan events otherwise.
Workflow: which tool has the highest error rate
This is the canonical "which tool errors most" question. Rank tools by error rate, but guard against small-sample noise with a HAVING floor on call volume:
posthog:execute-sql
SELECT
coalesce(nullIf(toString(properties.$mcp_exec_tool_call_name), ''), toString(properties.$mcp_tool_name)) AS tool,
count() AS total_calls,
countIf(toBool(properties.$mcp_is_error)) AS errors,
round(countIf(toBool(properties.$mcp_is_error)) * 100.0 / count(), 1) AS error_rate_pct
FROM events
WHERE event = '$mcp_tool_call'
AND coalesce(nullIf(toString(properties.$mcp_exec_tool_call_name), ''), toString(properties.$mcp_tool_name)) != ''
AND timestamp >= now() - INTERVAL 30 DAY
GROUP BY tool
HAVING total_calls >= 20
ORDER BY error_rate_pct DESC, total_calls DESC
LIMIT 20Report both rate and volume — a 100% error rate over 3 calls is rarely the real story; a 12% rate over 50,000 calls is. Offer to pull the top $mcp_error_message values for the worst tool (see below).
Workflow: tool-quality matrix
One row per tool with error rate, latency percentiles, and reach — mirrors the tool-quality screen. The ready-to-run query is in models-mcp.md under "Tool-quality matrix".
Workflow: why is a tool failing
For one tool's top failure buckets (grouped by harness), call posthog:query-mcp-tool-failures with the toolName — it's the typed equivalent of the query below. Failures come from the same source as the error rate: errored $mcp_tool_call events ($mcp_is_error), scoped by the effective tool name. Failures are grouped by $mcp_error_type (a semantic bucket: internal, validation, api_4xx, api_5xx, permission, timeout, rate_limited, missing_context) and the HTTP $mcp_error_status when present. To see individual errored calls inside a bucket — with the captured $mcp_error_message, session id, harness, and intent — pass the bucket's raw error_type/error_status to posthog:query-mcp-tool-failure-occurrences ($mcp_error_message is empty on events captured before message capture shipped):
posthog:execute-sql
SELECT
concat(
coalesce(nullIf(toString(properties.$mcp_error_type), ''), 'unknown'),
if(empty(coalesce(toString(properties.$mcp_error_status), '')), '',
concat(' (HTTP ', coalesce(toString(properties.$mcp_error_status), ''), ')'))
) AS failure,
count() AS n
FROM events
WHERE event = '$mcp_tool_call'
AND toBool(properties.$mcp_is_error)
AND coalesce(nullIf(toString(properties.$mcp_exec_tool_call_name), ''), toString(properties.$mcp_tool_name)) = '<tool>'
AND timestamp >= now() - INTERVAL 30 DAY
GROUP BY failure ORDER BY n DESC LIMIT 10$mcp_error_type is only populated on newer SDK/server paths — a chunk of errored calls carry neither type nor status and fall into the unknown bucket.
Workflow: slowest tools
Swap the aggregate for latency percentiles (quantile(0.95)(toFloat(properties.$mcp_duration_ms))) and order by p95_ms. The matrix query already returns p50_ms / p95_ms.
Constructing UI links
- Dashboard:
https://app.posthog.com/project/<project_id>/mcp-analytics/dashboard - Tool quality:
https://app.posthog.com/project/<project_id>/mcp-analytics/tool-quality
Always surface a UI link so the user can verify visually.
Tips
- Report error rate and call volume together; a
HAVING total_calls >= N
floor stops tools with very few calls from topping the list spuriously
- Exclude errored calls from latency percentiles only when asked — failed calls
are often the slow ones, and dropping them hides the problem
$mcp_client_namelets you cut quality by harness (Claude Code vs Cursor vs
…); the canonical bucketing multiIf is in models-mcp.md
- Harness bucketing is resolved server-side by
products/mcp_analytics/backend/mcp_harness.py — that's the source of truth, and posthog:query-mcp-harness-breakdown runs it. If your hand-written SQL disagrees with the screen, your bucketing has drifted from mcp_harness.py; prefer the typed tool over re-deriving it
Related skills
- `exploring-mcp-sessions` — drill into a
single agent run and its tool sequence
group agent goals and see which intents drive the errors
