dt-obs-analytics

Analytics — Dashboard & Notebook Query Extraction

A pipeline of three platform JavaScript scripts under scripts/ — two extractors feeding a shared analyzer runner:

scripts/extract-timeseries-dashboard.js ──┐
scripts/extract-timeseries-notebook.js  ──┴──► queryset.json ──► scripts/run-analyzer.js (any Davis analyzer)

Each script is invoked via:

dtctl exec function -f scripts/<script>.js --payload '<json>' -o json

-o json wraps the function return value under .result. When you need to inspect the result, read it directly from the output — no jq required. Pass the full raw output as queries to run-analyzer.js and it unwraps automatically.

Parsing a dashboard URL

When the entry point is a Dynatrace dashboard URL, extract the three components the scripts need:

https://<tenant>/ui/apps/dynatrace.dashboards/dashboard/<ID>#from=<from>&to=<to>&vfilter_<name>=<val>...

The timeframe in the URL fragment is the dashboard's display window. It is not injected into the extracted DQL — the extractor returns the DQL verbatim with its original $variable tokens and any embedded | timeframe clauses intact. Use the parsed from/to values only when calling run-analyzer.js to set the analysis window. If the user just wants to list the queries, the timeframe is informational only.

Note: when you pass run-analyzer.js an absolute timeframe.startTime (not a now... expression), it strips any embedded | timeframe ... stage from the DQL before analysis, so the analyzer honors your requested window rather than the query's baked-in one. For relative (now...) windows the embedded | timeframe is left intact. This means the query actually analyzed can differ from the extracted text — expected behavior, noted here so results line up with the window you asked for.

Quick bash parse (pure bash + sed/awk — no python needed):

DASHBOARD_URL="https://abc123.apps.dynatrace.com/ui/apps/dynatrace.dashboards/dashboard/5bea16c7-029b-43b6-9735-459db2d25bbf#from=2026-05-28T04%3A00Z&to=2026-05-28T05%3A00Z&vfilter_host_group=prod&vfilter_workload=my-svc"

# Minimal URL-decoder: turn %XX into \xXX and let printf interpret it.
urldecode() { local s="${1//+/ }"; printf '%b' "${s//%/\\x}"; }

DOC_ID=$(echo "$DASHBOARD_URL" | sed 's/#.*//' | awk -F/ '{print $NF}')
FROM=$(urldecode "$(echo "$DASHBOARD_URL" | sed -n 's/.*[#&]from=\([^&]*\).*/\1/p')")
TO=$(urldecode "$(echo "$DASHBOARD_URL"   | sed -n 's/.*[#&]to=\([^&]*\).*/\1/p')")
HOST_GROUP=$(echo "$DASHBOARD_URL" | sed -n 's/.*[#&]vfilter_host_group=\([^&]*\).*/\1/p')
WORKLOAD=$(echo "$DASHBOARD_URL"   | sed -n 's/.*[#&]vfilter_workload=\([^&]*\).*/\1/p')

For notebooks: path segment after /notebook/, or #share= value for /document/v0/#share=<ID> links.

Step 1 — Extract queries

From a dashboard

# All tiles
dtctl exec function -f scripts/extract-timeseries-dashboard.js \
  --payload '{"id":"<dashboard-id-or-name>"}' -o json

# Only tiles whose title matches a name the user mentioned (e.g. "CPU usage", "Kafka lag")
dtctl exec function -f scripts/extract-timeseries-dashboard.js \
  --payload '{"id":"<dashboard-id>","titleFilter":"CPU usage"}' -o json

When the user names a specific tile, chart, or section, pass its name as titleFilter rather than extracting the full dashboard. titleFilter is a case-insensitive substring or /regex/flags pattern. This keeps the queryset small and focused.

When it is not clear which tile(s) the user wants, do NOT extract all DQL — dashboards can have 20–50 tiles and returning all queries causes significant context bloat. Instead use a two-step flow:

1. List tile names with listOnly: true (no DQL, just titles):

   dtctl exec function -f scripts/extract-timeseries-dashboard.js \
     --payload '{"id":"<dashboard-id>","listOnly":true}' -o json
   # Returns: {"result":{"ok":true,"tiles":[{"id":"...","title":"CPU Usage","visualization":"lineChart"},...]}}

1. Show the tile names to the user and ask which tile(s) they mean.
2. Re-run with titleFilter for only the tile(s) of interest.

This avoids pulling 20–50 DQL queries into context when only 1–2 are relevant.

Payload knobs:

• id (required) — dashboard ID (UUID) or exact name. Preset IDs like my.dynatrace.infraops.preview.* work.
• titleFilter — case-insensitive substring ("CPU usage") or /regex/flags ("/^kafka/i").
• listOnly — when true, returns tiles: [{id, title, visualization}] without DQL. Use for disambiguation.
• compact — when true, returns only {id, title, dqlQuery} per tile (drops description, visualization, isTimeseries). Saves ~40% per-tile tokens. In listOnly mode, drops visualization too.
• includeSkipped — when true, returns the full skipped[] array. Default: only skippedCount is returned.

Response envelope:

{
  "ok": true,
  "documentId": "...", "documentName": "...", "documentVersion": 7,
  "queries": [
    { "id": "<tile-key>", "title": "...", "description": "...",
      "dqlQuery": "timeseries avg(dt.host.cpu.usage)",
      "visualization": "lineChart", "isTimeseries": true }
  ],
  "skipped": [{ "id": "...", "reason": "non-data tile (markdown)" }]
}

On failure: { "ok": false, "error": { "code": "...", "message": "..." } }.

From a notebook

Same envelope, different schema walk:

# All cells
dtctl exec function -f scripts/extract-timeseries-notebook.js \
  --payload '{"id":"<notebook-id-or-name>"}' -o json

# A specific section (if cell titles are set)
dtctl exec function -f scripts/extract-timeseries-notebook.js \
  --payload '{"id":"<notebook-id>","titleFilter":"JVM memory"}' -o json

Notebook cells often have empty titles — prefer addressing cells by id from the envelope if targeting a specific one.

Step 2 — Run an analyzer

Save the extractor output to a file, then pass it via shell substitution — the shell reads the file, so the JSON never enters the model's context:

# Run extractor, save output
dtctl exec function -f scripts/extract-timeseries-dashboard.js \
  --payload '{"id":"<id>","titleFilter":"CPU usage","compact":true}' -o json > queryset.json

# Shell substitution: $(cat queryset.json) is expanded by the shell, not the model
dtctl exec function -f scripts/run-analyzer.js \
  --payload '{
    "analyzerName": "dt.statistics.NoveltyScoreAnalyzer",
    "queries": '"$(cat queryset.json)"',
    "timeframe": { "startTime": "...", "endTime": "..." },
    "analyzerParams": { ... }
  }' -o json

run-analyzer.js unwraps the {"result":{...}} dtctl envelope automatically — pass the raw saved output as-is. No jq or parsing step is needed: normalization of the array / envelope / dtctl-output shapes happens inside the script.

For large querysets (many tiles), the inline $(cat ...) form can hit shell argument-length limits. Build the payload file and use dtctl's --data flag instead — still no jq and still out of model context:

{ printf '{"analyzerName":"dt.statistics.NoveltyScoreAnalyzer","timeframe":{"startTime":"now-1h","endTime":"now"},"queries":'
  cat queryset.json
  printf '}'; } > payload.json

dtctl exec function -f scripts/run-analyzer.js --data payload.json -o json

Key payload knobs for run-analyzer.js:

• minScore — drop results below this threshold (e.g. 0.5). Auto-detects score field from noveltyScore, anomalyScore, correlationCoefficient, correlation, coefficient. Pass scoreField to override.
• scoreField — explicit field name to read score from (e.g. "noveltyScore").

The queries field accepts any of: a raw array, the extractor envelope ({queries:[...]}), or the full dtctl output ({"result":{"queries":[...]}}). All three are normalized automatically.

Common analyzers

Correlation mode

Pass metricQuery to correlate every query in the set against a single primary DQL string:

dtctl exec function -f scripts/run-analyzer.js \
  --payload '{
    "analyzerName": "dt.statistics.SimplePearsonCorrelationAnalyzer",
    "queries": '"$(cat queryset.json)"',
    "metricQuery": "<dqlQuery of the primary tile, copied from extractor output>",
    "timeframe": { "startTime": "...", "endTime": "..." }
  }' -o json

When chaining from a previous analyzer run (e.g. anomaly detection → correlation), use metricQueryFrom instead. The script picks the highest-scored result's dqlQuery automatically:

dtctl exec function -f scripts/run-analyzer.js \
  --payload '{
    "analyzerName": "dt.statistics.SimplePearsonCorrelationAnalyzer",
    "queries": '"$(cat queryset.json)"',
    "metricQueryFrom": '"$(cat findings.json)"',
    "timeframe": { "startTime": "...", "endTime": "..." }
  }' -o json

metricQuery takes precedence if both are set.

Variable substitution

Dashboard queries often contain $variable tokens (from URL vfilter_* params). Pass them via variables to substitute before execution:

dtctl exec function -f scripts/run-analyzer.js \
  --payload '{
    "analyzerName": "dt.statistics.anomaly_detection.SeasonalBaselineAnomalyDetectionAnalyzer",
    "queries": [...],
    "timeframe": { "startTime": "2026-05-28T04:00Z", "endTime": "2026-05-28T05:00Z" },
    "variables": { "host_group": "prod", "workload": "my-svc" }
  }' -o json

Build variables from vfilter_* URL params by stripping the vfilter_ prefix. Trailing * wildcards are stripped automatically. Unresolved tokens are cleaned up from DQL filter clauses rather than left to error.

Response shape

{
  "ok": true,
  "checkedAt": "...",
  "analyzerName": "...",
  "summary": { "checked": 12, "completed": 11, "errors": 1 },
  "results": [
    {
      "id": "tile-key", "title": "CPU usage", "dqlQuery": "...",
      "output": <raw analyzer output>,
      "executionStatus": "COMPLETED"
    }
  ],
  "errors": [ { "id": "...", "error": "..." } ]
}

The output field is the raw analyzer result. Interpret it based on the analyzer:

• Anomaly detection: look for anomalyScore, anomalies[], or raisedAlerts[] in each output entry. Score ≥ 0.7 → abnormal, ≥ 0.4 → borderline.
• Novelty: look for noveltyScore (or the closest score-like numeric field). Score ≥ 0.7 → novel.
• Correlation: look for correlationCoefficient. Sort by |correlationCoefficient| descending; drop entries where |correlationCoefficient| < 0.5.

End-to-end: "what's abnormal on this dashboard?"

# 1. Extract queries — shell reads file, JSON stays out of model context
dtctl exec function -f scripts/extract-timeseries-dashboard.js \
  --payload '{"id":"5bea16c7-029b-43b6-9735-459db2d25bbf","compact":true}' \
  -o json > queryset.json

# 2. Run anomaly detection — $(cat queryset.json) expanded by shell, not model
dtctl exec function -f scripts/run-analyzer.js \
  --payload '{
    "analyzerName": "dt.statistics.anomaly_detection.SeasonalBaselineAnomalyDetectionAnalyzer",
    "queries": '"$(cat queryset.json)"',
    "timeframe": { "startTime": "2026-05-28T04:00Z", "endTime": "2026-05-28T05:00Z" },
    "variables": { "host_group": "prod", "workload": "my-svc" }
  }' -o json > findings.json

# 3. Correlate — metricQueryFrom picks the top finding automatically
dtctl exec function -f scripts/run-analyzer.js \
  --payload '{
    "analyzerName": "dt.statistics.SimplePearsonCorrelationAnalyzer",
    "queries": '"$(cat queryset.json)"',
    "metricQueryFrom": '"$(cat findings.json)"',
    "timeframe": { "startTime": "2026-05-28T04:00Z", "endTime": "2026-05-28T05:00Z" }
  }' -o json

Verifying a single extracted query

Read the dqlQuery field from the extractor output and pass it directly:

dtctl query --query "<dqlQuery copied from extractor output>" -o json | head -40

Gotchas

• Never read raw dashboard JSON yourself. dtctl get dashboard <id> -o json is typically 50–200 KB. The extractor reads it on the platform and returns a compact envelope (~5–15 KB).
• Never extract all tiles when only one is needed. A 50-tile dashboard returns 50 DQL queries into context. If the user names a tile, use titleFilter. If it's ambiguous, use listOnly: true first to ask which tile — then extract only that one.
• Variables are required for filtered dashboards. Queries with unsubstituted $variable tokens silently drop entity filters (e.g. in(field, $undefined) evaluates to true). Always pass variables when the URL has vfilter_* params.
• Schema drift. If a tile lands in skipped with reason no DQL query found, the dashboard schema has a query location the extractor doesn't know about — add it to the pickQuery candidate list in the script.
• Analyzer availability. Not all Davis analyzers are available on every tenant. If a call comes back with Could not find an analyzer with name '...' or is not a function, list what's actually registered: dtctl get analyzers -o json.
• Statistical fallback removed. run-analyzer.js only calls Davis analyzers. For historical anomaly detection, pass a long trainingTimeframe via analyzerParams (e.g. { "trainingTimeframe": { "startTime": "now-30d", "endTime": "now-1d" } }), or query the DQL directly.
• Comments in queries. Queries starting with // comment lines are classified correctly by the extractor (leading line/block comments are stripped before the timeseries check).

Scripts reference

• scripts/extract-timeseries-dashboard.js — extracts timeseries DQL from a dashboard
• scripts/extract-timeseries-notebook.js — same for notebooks
• scripts/run-analyzer.js — generic Davis analyzer runner