oodle-monitors

Documento original del Skill

Contenido del repositorio de origen con títulos, ejemplos, código, tablas, enlaces e imágenes preservados.

Oodle Monitors — CRUD and Alerting

This skill teaches the agent to build, validate, and update Oodle monitors so that alerts are actionable, scoped, and free from flapping.

Prerequisites

bash

# Install + configure (see oodle-cli skill)
brew install oodle-ai/oodle/oodle
oodle configure
# or
export OODLE_API_KEY=<key>
export OODLE_INSTANCE=<instance>
export OODLE_DEPLOYMENT=<url>

Confirm authentication and that at least one monitor list call succeeds before creating new monitors:

bash

oodle monitors list -o json | jq 'length'

Command Execution Order

Before running any oodle command:

Check whether the required resource ID or name is already in context.
If not, run the discovery command (e.g., oodle monitors list -o json).
If the result is ambiguous, ask the user to confirm before proceeding.
Run the target command with the resolved ID.
Do not run speculative commands (e.g., do not delete without first get-ing the resource).

Quick Reference

Task	Command
List all monitors	`oodle monitors list -o json`
Filter by status	`oodle monitors list --status alert -o json`
Filter by labels	`oodle monitors list --labels env=prod,team=platform -o json`
Get one monitor	`oodle monitors get <id> -o json`
Create from file	`oodle monitors create -f monitor.json`
Update from file	`oodle monitors update <id> -f monitor.json`
Delete (CI)	`oodle monitors delete <id> --force`

Common Operations

Listing monitors

bash

# ✅ CORRECT — JSON output for scripting
oodle monitors list -o json

# ✅ CORRECT — narrow with --status to find only firing alerts
oodle monitors list --status alert -o json

# ✅ CORRECT — narrow with --labels for a specific team
oodle monitors list --labels env=prod,team=platform -o json

# ❌ WRONG — pulling everything then grepping
oodle monitors list | grep CPU

Reading a monitor before changing it

bash

# ✅ CORRECT — fetch full JSON, edit, then update
oodle monitors get mon_123 -o yaml > monitor.yaml
$EDITOR monitor.yaml
oodle monitors update mon_123 -f monitor.yaml

# ❌ WRONG — building update payload from memory; overwrites unrelated fields
oodle monitors update mon_123 -f <(echo '{"options":{"thresholds":{"critical":90}}}')

Creating a monitor

A complete, valid monitor JSON:

json

{
  "name": "High CPU on web servers",
  "type": "metric alert",
  "query": "avg(last_5m):avg:system.cpu.user{env:prod,service:api} by {host} > 80",
  "message": "CPU above 80% on {{host.name}}. Runbook: https://runbooks.example.com/cpu\n@slack-ops",
  "labels": {"team": "platform", "env": "prod"},
  "options": {
    "thresholds": {
      "critical": 80,
      "critical_recovery": 70,
      "warning": 60,
      "warning_recovery": 50
    }
  }
}

bash

# ✅ CORRECT
oodle monitors create -f monitor.json

# ❌ WRONG — no `type`, no `options.thresholds`, monitor will be rejected
oodle monitors create -f <(echo '{"name":"x","query":"y"}')

Updating a monitor

bash

# ✅ CORRECT — get → edit → update
oodle monitors get mon_123 -o json > monitor.json
jq '.options.thresholds.critical = 85' monitor.json > monitor.new.json
oodle monitors update mon_123 -f monitor.new.json

# ❌ WRONG — sending only the changed field; missing fields become null
oodle monitors update mon_123 -f <(echo '{"options":{"thresholds":{"critical":85}}}')

Deleting a monitor

bash

# ✅ CORRECT — verify first, then delete
oodle monitors get mon_123 -o json > /dev/null
oodle monitors delete mon_123 --force

# ❌ WRONG — speculative delete by name match
oodle monitors delete "$(oodle monitors list | grep CPU | head -1 | awk '{print $1}')" --force

Best Practices

Use a `last_5m` (or longer) evaluation window, not `last_1m`

Short windows cause alert flapping on normal traffic spikes. Use last_5m minimum for production alerts, last_15m for noisy metrics.

bash

# ✅ CORRECT
"query": "avg(last_5m):avg:system.cpu.user{env:prod} by {host} > 80"

# ❌ WRONG — flaps on every brief spike
"query": "avg(last_1m):avg:system.cpu.user{env:prod} by {host} > 80"

Scope queries with explicit labels — never use `{*}`

{*} matches every series in the system and produces alerts for resources you don't own.

bash

# ✅ CORRECT — scoped to a specific env + service
"query": "avg(last_5m):avg:system.cpu.user{env:prod,service:api} by {host} > 80"

# ❌ WRONG — alerts on every host in the org
"query": "avg(last_5m):avg:system.cpu.user{*} > 80"

Always set `*_recovery` thresholds below the trigger thresholds

Without recovery thresholds the monitor stays in alert state until the metric drops below the critical threshold exactly — small oscillations keep the alert active forever.

bash

# ✅ CORRECT — clear recovery band (10pt below trigger)
"thresholds": {"critical": 80, "critical_recovery": 70, "warning": 60, "warning_recovery": 50}

# ❌ WRONG — no recovery values; monitor never cleanly recovers
"thresholds": {"critical": 80, "warning": 60}

Put a runbook URL and an `@notifier` handle in `message`

Alerts without an action are noise. Every monitor message must answer "what do I do?" and "who is paged?".

bash

# ✅ CORRECT
"message": "CPU above 80% on {{host.name}} (env=prod, service=api).\nRunbook: https://runbooks.example.com/cpu\n@slack-ops @pagerduty-platform"

# ❌ WRONG — no actionable content, no routing
"message": "CPU is high"

Tag every monitor with at least `team` and `env` labels

Labels are how notification policies route alerts and how oodle monitors list --labels ... filters work.

bash

# ✅ CORRECT
"labels": {"team": "platform", "env": "prod", "service": "api"}

# ❌ WRONG
"labels": {}

Failure Handling

Error	Cause	Fix
401 Unauthorized	Invalid or missing API key	Run `oodle configure` or set `OODLE_API_KEY`
404 Not Found	Monitor ID does not exist	Verify with `oodle monitors list -o json`
connection refused	Wrong `OODLE_DEPLOYMENT` URL	Check `OODLE_DEPLOYMENT` env var
`invalid query`	PromQL/Datadog-style query has a syntax error	Test the query in the UI metrics explorer; ensure `avg(last_5m):` prefix and `by {label}` suffix
Alert never fires	Query returns no data, or threshold is unreachable	Run `oodle metrics list --match <metric>` to confirm the metric exists; lower the threshold temporarily and re-check
Too many alerts (flapping)	Evaluation window too short, missing recovery thresholds	Increase window to `last_5m` or `last_15m`; set `critical_recovery` and `warning_recovery`
`no data` alerts	Agent not reporting, or label filter excludes all hosts	Verify the agent is alive (`oodle metrics list --match up`); widen the label filter to confirm any series match
429 Too Many Requests	Bulk monitor creation hit rate limit	Add `--retries 3`, throttle to <10 creates per second

Oodle Monitors — CRUD and Alerting

Prerequisites

Command Execution Order

Quick Reference

Common Operations

Listing monitors

Reading a monitor before changing it

Creating a monitor

Updating a monitor

Deleting a monitor

Best Practices

Use a `last_5m` (or longer) evaluation window, not `last_1m`

Scope queries with explicit labels — never use `{*}`

Always set `*_recovery` thresholds below the trigger thresholds

Put a runbook URL and an `@notifier` handle in `message`

Tag every monitor with at least `team` and `env` labels

Failure Handling

References

Más Skills

oodle-alerting

oodle-dashboards

oodle-drop-rules

oodle-onboarding

oodle-monitors

Oodle Monitors — CRUD and Alerting

Prerequisites

Command Execution Order

Quick Reference

Common Operations

Listing monitors

Reading a monitor before changing it

Creating a monitor

Updating a monitor

Deleting a monitor

Best Practices

Use a last_5m (or longer) evaluation window, not last_1m

Scope queries with explicit labels — never use {*}

Always set *_recovery thresholds below the trigger thresholds

Put a runbook URL and an @notifier handle in message

Tag every monitor with at least team and env labels

Failure Handling

References

Más Skills

oodle-alerting

oodle-dashboards

oodle-drop-rules

oodle-onboarding

Use a `last_5m` (or longer) evaluation window, not `last_1m`

Scope queries with explicit labels — never use `{*}`

Always set `*_recovery` thresholds below the trigger thresholds

Put a runbook URL and an `@notifier` handle in `message`

Tag every monitor with at least `team` and `env` labels