aws/agent-toolkit-for-aws

setting-up-cloudwatch-observability

First-time setup for CloudWatch observability — Omni (CloudWatch Application Observability) and classic CloudWatch.

Ver código-fonte

Como usar

  1. Copie o comando e verifique o pacote antes de instalar no seu ambiente de agente.
  2. Consulte no original as ferramentas, credenciais e ambientes necessários.
  3. Descreva os dados de entrada e a entrega esperada; confira a saída com as fontes.
npx skills add https://github.com/aws/agent-toolkit-for-aws --skill setting-up-cloudwatch-observability

Antes de começar

  • A instalação não inclui assinatura de modelo, franquia API ou acesso aos seus arquivos.
  • Estrelas e instalações indicam atividade, não um teste independente de qualidade ou segurança.
Documento original do Skill

Renderizado do repositório de origem, preservando títulos, exemplos, código, tabelas, links e imagens.

Setting Up CloudWatch Observability

Scope: First-time CloudWatch observability setup, for both products — a CloudWatch Application Observability (Omni) Space from creation through first traces flowing, and onboarding a service to classic CloudWatch Application Signals. For using what is already set up (queries, dashboards, alarms, Omni alerts, evaluations, live debugging), route to aws-observability.

Two products, two reference folders

This skill owns setup for two products that share the CloudWatch name but are separate services. The folder a reference lives in is the signal for which product it belongs to — the same convention aws-observability uses:

FolderProductWhat setup means there
references/cloudwatch-omni/CloudWatch Omni (Application / Agent Observability)Domain → Space → grants → telemetry in → plain ADOT SDK instrumentation. No add-on, no CloudWatch Agent application_signals config, no port 4316.
references/cloudwatch/Classic CloudWatchOnboarding a service to Application Signals: ADOT auto-instrumentation, the amazon-cloudwatch-observability EKS add-on, the CloudWatch Agent, monitored service, ServiceEvents, CI/CD git/deployment metadata, port 4316.

The two instrumentation paths are mutually exclusive on a given workload — the Omni path explicitly forbids the Application Signals env vars and the add-on, and vice versa. Decide the product first (Routing Rules 1–4 below), then stay inside that folder. Enabling one does not replace the other; an account may run both, on different services.

Works best with the AWS MCP server — enables running AWS CLI commands directly. All guidance also works with standard AWS CLI access (aws cloudwatchomni ... on the Omni path; aws eks, aws iam, and aws application-signals on the Application Signals path).

Concepts — Omni (references/cloudwatch-omni/)

These are Omni's resources. None of them exist on the Application Signals path — that one has no Domain, Space, grant, or Dataset; its unit is a monitored service, and access is plain IAM. Do not ask an Application Signals onboarding customer for a spaceId.

TermWhat it is
DomainThe identity boundary. Carries the authorization provider (IAM or Identity Center) and owns the endpoint URL customers reach Omni through. One per account, or one shared across an AWS Organization.
SpaceA workspace holding telemetry, in exactly one account and one Region. Created under a Domain. At most one per account per Region. Region rule: under an IAM-only Domain a Space may sit in a Region other than the Domain's; under an account-scoped Identity Center Domain the Space must be in the Domain's own Region; under an org-scoped Identity Center Domain it must be in a Region the instance is replicated to.
Access grantAttaches a principal — person, group, IAM identity, or async workload — to one Space at a permission level. The only way anyone reaches data through a Space — it does not restrict the source CloudWatch log groups, which stay readable under their own IAM.
Access ProfileA named boundary for async workloads (alerts, integrations, agents) that act without a person in the loop. It is only a named container: create-access-profile takes a Space, a name, and a description and nothing else — no permission, action, or scope input. It does something only once two separate sets of grants exist (what the profile may do; which workloads may assume it) and a workload names it.
DatasetWhat queries run against. Telemetry arrives through the CloudWatch OTLP endpoints, or by forwarding what is already in CloudWatch log groups.

Omni setup order from nothing: Domain → Space → grants → telemetry in → instrumentation. (The Application Signals equivalent is much shorter and has no prerequisite resources — add-on/agent, IAM, then the per-platform enablement change; see references/cloudwatch/application-signals-onboarding.md.) "Telemetry in" (a collector exporting to CloudWatch's OTLP endpoints, plus dataset forwarding for what is already in CloudWatch) is its own step, separate from instrumenting the workloads. Access Profiles are conditional, not a step in the sequence — only when async workloads (alerts, integrations, agents) are involved. Whenever you give this sequence, also say that instrumentation or forwarding started before a Space exists appears to succeed while delivering telemetry nowhere the customer can see — a customer who checks Omni first reads a working, empty Space as a failure. For the concept relationships and the full arc, see references/cloudwatch-omni/app-basics.md.

This is a routing skill. Classify the user's setup request and delegate to the correct reference. Omni references live under references/cloudwatch-omni/; classic-CloudWatch (Application Signals) references live under references/cloudwatch/.

User intentReference
Onboard a service to Application Signals (auto-instrumentation, the amazon-cloudwatch-observability EKS add-on, CloudWatch Agent IAM for Application Signals, monitored service, reporting telemetry to Application Signals, ServiceEvents, the two onboarding tiers)references/cloudwatch/application-signals-onboarding.md
Propagate ServiceEvents git/deployment metadata through CI/CD (the 5 OTEL_AWS_SERVICE_EVENTS_* vars, per-provider patterns)references/cloudwatch/application-signals-cicd-metadata.md
Per-platform × per-language Application Signals enablement steps once platform and language are knownThe matching references/cloudwatch/appsignals-guides/<platform>-<language>.md (e.g. references/cloudwatch/appsignals-guides/eks-python.md)
Turn on Dynamic Instrumentation for a service at onboarding time (the OTEL_AWS_DYNAMIC_INSTRUMENTATION_* vars and their IAM)references/cloudwatch/application-signals-onboarding.md (Step 5d). Using it to debug is aws-observability
Understand what Omni is, its concepts, or where to startreferences/cloudwatch-omni/app-basics.md
Instrument an AI agent (ADOT, OpenInference, framework detection, trace verification), or deploy an agent to production and get traces flowing to CloudWatch — env vars per platform (AgentCore, Lambda, or other platforms such as ECS/EC2/EKS), routing spans to a custom trace log group, IAM permissions needed, how the export authenticates (AWS credentials, no API token), ADOT version requirementsreferences/cloudwatch-omni/omni-agents-instrumentation/omni-agents-instrumentation.md (§ Production deployment for the deploy case)
Per-framework OpenInference guide (LangChain, LangGraph, Strands, CrewAI, OpenAI Agents, Vercel AI) once the agent framework is knownreferences/cloudwatch-omni/omni-agents-instrumentation/openinference-framework-guide.md, then the matching references/cloudwatch-omni/omni-agents-instrumentation/instrument-<framework>.md
Instrument an application (ADOT SDK on EC2/ECS/EKS/Lambda — Python, Node.js, Java, .NET)references/cloudwatch-omni/instrumentation/instrumentation.md
Monitor a vLLM model server on ECS or EKS so it appears on Omni's AI Inference page — engine metrics, GPU (DCGM) metrics, vLLM traces, engine configuration — including KServe and Knative; or an inference service missing from that page, or its GPU panels emptyreferences/cloudwatch-omni/instrumentation/ai-inference.md
Emit a custom application or agent metric so it is queryable in Omni (why OTLP and not PutMetricData/EMF)references/cloudwatch-omni/instrumentation/instrumentation.md (§ Custom metrics)
Create an account-scoped Space or Domainreferences/cloudwatch-omni/spaces-and-domains.md
Whether a Space can be in a different Region from its Domainreferences/cloudwatch-omni/spaces-and-domains.md (Prerequisites → Region rules)
The AgentCore evaluation role that create-space asks for — what it is, whether to create onereferences/cloudwatch-omni/spaces-and-domains.md (Step 3 → Also resolve the AgentCore evaluation role)
Let Omni run Bedrock models for a Space (Agent Playground, evaluations run from Omni), or a Bedrock AccessDeniedException from those featuresreferences/cloudwatch-omni/spaces-and-domains.md (Step 3 → Optional: let Omni invoke Bedrock models)
A Space that was created successfully but returns an authorization error when usedreferences/cloudwatch-omni/spaces-and-domains.md (Troubleshooting → If the Space was created but cannot be used) — a space access role trust-policy problem, not a grant problem
Create a Domain shared across an AWS Organizationreferences/cloudwatch-omni/org-domains.md
Configure access grants for people or IAM identities or alertsreferences/cloudwatch-omni/access-grants.md
An aws cloudwatchomni command reporting that the service is not supported, or a request to repair the customer's CLI or SDK so Omni commands workNot a setup task — a local client-version problem. STOP and route to aws-observability
A programmatic caller denied on a Space it already had working — a CI role, SDK client, CloudFormation role, coding agent, or a workload under an Access Profilereferences/cloudwatch-omni/access-grants.md (Grants, IAM, and Access Profiles), and aws-observability for the API-side view
Bound an alert, integration, or agent with an Access Profilereferences/cloudwatch-omni/access-profiles.md
Deploy an OTel Collector so an instrumented app has somewhere to export to (EC2/ECS/EKS) — it exports to CloudWatch's own per-signal OTLP endpointsreferences/cloudwatch-omni/instrumentation/collector.md
An OTLP collector already running (a CloudWatch Agent with an opentelemetry section or OTLP on 4317/4318, or otelcol-contrib) whose telemetry does not arrive, in whole or for one signal such as tracesreferences/cloudwatch-omni/instrumentation/collector.md (Verify), then the platform collector guide it links for the troubleshooting table
Enable Transaction Search so traces reach a Space (spans land in aws/spans only once it is on — per account, per Region)references/cloudwatch-omni/instrumentation/collector.md (Step 1)
Forward telemetry already in CloudWatch into the Datasetreferences/cloudwatch-omni/data-forwarding-and-centralization.md
Send your application's own telemetry from Azure (the logs/metrics/traces your service emits, via the CloudWatch agent on an Azure VM/AKS)references/cloudwatch-omni/azure-ingestion/azure-ingestion.md (intent triage), then references/cloudwatch-omni/azure-ingestion/custom-telemetry.md (the CloudWatch-agent-on-VM/AKS procedure)
Ingest telemetry your Azure resources emit on their own — resource and platform diagnostic logs and metrics, VNet flow logs — by connecting an Azure tenant (CloudWatch Connector, telemetry manifest, pipeline); also an AccessDeniedException on a connector, manifest, or pipeline operation, a request for the Bicep, `az` or setup-script commands for an Azure connector, the Azure tenant trust / outbound federation setup, the telemetry manifest or scope, or how the Azure side authenticates to CloudWatchreferences/cloudwatch-omni/azure-ingestion/azure-ingestion.md (intent triage), then references/cloudwatch-omni/azure-ingestion/vended-telemetry.md (the connector/manifest/pipeline path)
Send your application's own telemetry from Google Cloud (the logs/metrics/traces your service emits, via the CloudWatch agent on a GCE instance/GKE cluster)references/cloudwatch-omni/gcp-ingestion/gcp-ingestion.md (intent triage), then references/cloudwatch-omni/gcp-ingestion/custom-telemetry.md (the CloudWatch-agent-on-GCE/GKE procedure)
Connect Slack to a Space for the first timereferences/cloudwatch-omni/slack-integration.md
Connect the AWS DevOps Agent to a Space (Settings → Integrations → AWS DevOps Agent), an empty agent-space picker, or Investigations reporting the AWS DevOps Agent is not connectedreferences/cloudwatch-omni/devops-agent-integration.md. Creating the DevOps Agent space itself and running or chatting with investigations belong to the aws-agents-for-devsecops plugin
Whether a custom MCP tool server can be registered (Omni has none)references/cloudwatch-omni/custom-mcp-integration.md

Routing Rules

  1. If the user is asking what Omni is, what a Domain or Space or grant means, where to start, or the end-to-end order of steps from nothing — rather than asking to perform one setup step — route to references/cloudwatch-omni/app-basics.md and answer the sequence from its "Setup order" section (not from a single procedure file such as collector.md, which covers one step). It also carries the boundary against aws-observability. A question about one concept's rules — a Space's Region relative to its Domain, the roles create-space needs — belongs to the procedure file for that concept (spaces-and-domains.md), which the routing table names.
  2. If the request is about instrumenting an AI agent project (adding OTel to a framework such as LangChain, LangGraph, Strands, CrewAI, OpenAI Agents, or Vercel AI), it is Omni — route to references/cloudwatch-omni/omni-agents-instrumentation/omni-agents-instrumentation.md. There is no Application Signals path for an agent framework. Likewise, monitoring a vLLM model server (its engine and GPU metrics, or a service missing from Omni's AI Inference page) is Omni: route to references/cloudwatch-omni/instrumentation/ai-inference.md. It is not ADOT SDK instrumentation, so rule 3 does not apply.
  3. If the request is about instrumenting a general application (a service on EC2, ECS, EKS, or Lambda in Python, Node.js, Java, or .NET — not an AI-agent framework), decide the product before opening any procedure file — the two paths are mutually exclusive on a workload:
  • Names Application Signals, ServiceEvents, a "monitored service", the service map, the amazon-cloudwatch-observability add-on, the CloudWatch Agent's application_signals configuration, or port 4316 → the CloudWatch path: references/cloudwatch/application-signals-onboarding.md, then the matching references/cloudwatch/appsignals-guides/<platform>-<language>.md.
  • Names Omni, Application Observability, a Space, a Domain, or a Dataset — or explicitly rules Application Signals out ("plain OTel", "no add-on", "no agent sidecar") → the Omni path: references/cloudwatch-omni/instrumentation/instrumentation.md.
  • Naming the CloudWatch Agent alone decides nothing: it is also the Omni path's default collector. An agent with an opentelemetry section or OTLP receivers on 4317/4318 is the Omni collector, and its delivery problems belong to rule 6.
  • Names neither → rule 4.
  1. If an instrumentation, ADOT, or collector request names neither product (neither Omni or a Space, nor Application Signals, ServiceEvents, or the amazon-cloudwatch-observability add-on), probe for a Space in the target Region before choosing. list-spaces is account-global: the --region flag only selects the endpoint, and the response lists every Space in the account, each with its own region, so filter to the target Region rather than trusting a non-empty list: aws cloudwatchomni list-spaces --region <region> --query "items[?region=='<region>']". A Space in that Region → the Omni path, references/cloudwatch-omni/instrumentation/instrumentation.md. An empty filtered list → the customer has not adopted Omni in that Region, so Application Signals is the live product for them: references/cloudwatch/application-signals-onboarding.md. Either way the request stays in this skill — the probe picks the folder, not the skill. If the probe errors with an unknown service, that is the CLI model, not evidence Omni is absent — fall back to the customer's wording and ask only if still inconclusive.
  2. If the request is about Space/Domain creation or configuration, route to the matching setup reference. An authorization error that appears the moment a just-created Space is used is part of this — it is a space access role trust-policy problem (create-space never verifies the role can be assumed), so route to references/cloudwatch-omni/spaces-and-domains.md → "If the Space was created but cannot be used", not to the access-grants reference.
  3. If the user asks about getting telemetry to a destination — telemetry not yet reaching CloudWatch — deploy an OTel Collector on EC2/ECS/EKS so an instrumented workload has somewhere to send OTLP, and wire the app's OTEL_EXPORTER_OTLP_ENDPOINT to it: references/cloudwatch-omni/instrumentation/collector.md. The same file covers an OTLP collector that is already deployed but whose telemetry does not arrive, in whole or for one signal (its Verify section and the platform guide's troubleshooting table). The collector exports straight to CloudWatch's own per-signal OTLP endpoints. If instead the telemetry is already in CloudWatch log groups and needs forwarding into the Dataset, route to references/cloudwatch-omni/data-forwarding-and-centralization.md. Both the collector's OTLP export and CloudWatch's OTLP endpoints authenticate with SigV4 (AWS credentials), including OIDC-federated credentials for a workload outside AWS (see references/cloudwatch-omni/azure-ingestion/custom-telemetry.md or references/cloudwatch-omni/gcp-ingestion/custom-telemetry.md). A sender that cannot obtain AWS credentials can send OTLP metrics with a CloudWatch API key (bearer token, the collector's bearertokenauth extension): see the aws-observability skill, its OTel metrics reference (Ingestion section). A token-based path for logs and traces is not covered by these skills; do not improvise one, and say so plainly. Telemetry a cloud's resources emit on their own is pulled, not sent, so an authentication question about it has a wrong premise. On Azure — resource and platform diagnostic logs, VNet flow logs — the resources write into the customer's own tenant: diagnostic settings to Event Hubs, VNet flow logs to Storage. AWS then reads from those using the CloudWatch Connector's federated identity, a telemetry manifest declares which resources to read from, and a pipeline writes what it reads into a log group in the customer's own account. Nothing in Azure holds or needs an AWS credential, so no token, API-key or shared-key ingestion endpoint exists for it. Give that flow before anything else. Full path, including setup and diagnosis: references/cloudwatch-omni/azure-ingestion/vended-telemetry.md.
  1. If the request is about getting Azure or Google Cloud telemetry into CloudWatch, route by cloud — references/cloudwatch-omni/azure-ingestion/azure-ingestion.md for Azure, references/cloudwatch-omni/gcp-ingestion/gcp-ingestion.md for Google Cloud — and decide by intent, because the two clouds differ in what is supported. If the customer wants the telemetry their own application produces (the logs/metrics/traces from their code), that is supported on both via the CloudWatch agent on an Azure VM or AKS cluster, or on a GCE instance or GKE cluster — follow the reference. If they want telemetry their cloud's resources emit on their own (the equivalent of AWS VPC flow logs / Route 53 logs), the answer splits: for Azure it is supported by its own mechanism — a CloudWatch Connector, a telemetry manifest, and a pipeline — so route to references/cloudwatch-omni/azure-ingestion/vended-telemetry.md; for Google Cloud (Cloud Audit Logs, the VPC flow logs Google emits) it is not supported, so say so and do not attempt a setup. Do not generalise either answer to the other cloud. If the request names no cloud, ask which one before giving any steps.
  2. If the user asks to connect, enable, or authorize Slack for a Space for the first time (including granting the operator permission to use it), route to references/cloudwatch-omni/slack-integration.md. Using Slack after it is connected (posting findings to a channel, mentioning the assistant, or searching Slack) happens in Slack and the console and is not covered by these skills; Slack as an alert notification target is covered by aws-observability's Omni alerts reference.
  3. Application Signals, ServiceEvents, and Dynamic Instrumentation are CloudWatch features, and they split by setup versus use — not by skill. Enabling them on a service that does not have them yet is setup and belongs here, under references/cloudwatch/: Application Signals onboarding and ServiceEvents in references/cloudwatch/application-signals-onboarding.md (with the CI/CD metadata chain in references/cloudwatch/application-signals-cicd-metadata.md), and switching Dynamic Instrumentation on at instrumentation time in that same file's Step 5d. They must never appear in an Omni instrumentation change — for that they remain out of scope, and rule 3 is how you tell the two paths apart. Using them once the service reports — reading the service map, alarming on Application Signals metrics, or placing breakpoints and reading snapshots with Dynamic Instrumentation — is day-to-day work: STOP and route to aws-observability.
  4. If the user already has a working Space, or a service already reporting to Application Signals, and is asking about queries/dashboards/alarms/alerts/investigation, STOP and route to aws-observability.
  5. If the user asks to register their own MCP tool server, connect a custom MCP server, or set up MCP authentication: CloudWatch Omni does not let customers add a custom MCP server. There is no console screen for it and the integration type is not enabled for customer use, so the attempt is refused. Say so plainly and stop; do not describe a setup flow, and do not offer a substitute. See references/cloudwatch-omni/custom-mcp-integration.md.
  6. If the user asks to connect GitHub, link a GitHub organization or repositories, or enrich the application map from source code: CloudWatch Omni has no GitHub integration. Say so plainly and stop — do not walk through a setup. Do not offer a substitute: the Application Signals GitHub Action and the CloudWatch Logs GitHub audit-log source are separate CloudWatch features, not a way to connect GitHub to Omni, and a custom MCP server is not a GitHub integration. A GitHub-related feature found in public documentation is not evidence of an Omni GitHub integration.
  7. If the request names neither product — "add observability to my API", "how should I begin monitoring my workload", with no mention of Omni, a Space, a Domain, or of a CloudWatch feature — do not assume either one. Which product a customer wants to set up is their intent, not something the account's current state tells you, so ask, framed product versus product: CloudWatch Omni (Application/Agent Observability — Domains, Spaces, grants, a Dataset) or CloudWatch (log groups, alarms, Log Insights, Application Signals). Both answers are served here, from different reference folders, so this is a question about which product to set up — not about which skill to use, and not a reason to hand the request to aws-observability. In the same message, explain the boundary so they can choose — it is setup versus use: creating or configuring a Domain, Space, grant, or Access Profile, getting telemetry flowing for the first time, instrumenting an application or agent for Omni, and onboarding a service to Application Signals are all first-time setup and belong here; querying telemetry, building dashboards, configuring alarms or Omni alerts, investigating a live problem, and debugging with Dynamic Instrumentation are day-to-day use and belong to aws-observability. Do not start walking through Domain/Space creation or Application Signals onboarding until they confirm which product; having asked, stop and wait.
  8. If the user asks to connect the AWS DevOps Agent to a Space, or the connect card or Investigations shows it is not connected, route to references/cloudwatch-omni/devops-agent-integration.md. These skills cover only that Omni-side connection. Creating or configuring the DevOps Agent space, and running, following, or chatting with its investigations from a coding agent, belong to the agent's dedicated `aws-agents-for-devsecops` plugin; point the customer there rather than reconstructing that workflow.
do mesmo repositório

Mais Skills

Todos os Skills
aws
Oficial

deploying-containers-to-amazon-lightsail

Deploys, hosts, or runs a container or Docker app on Amazon Lightsail Container Service and returns a live, managed HTTPS URL at a flat monthly price — no cluster, VPC, load balancer, or registry to set up. Used when deploying a container image to Lightsail from a local Docker build, a public registry image, or a GitHub repo, or when a Lightsail container deployment fails or won't serve — including when the request doesn't name Lightsail but asks to run one always-on container behind a public UR

Instalações / 1h
4
GitHub Stars
3,5 mil
Atualizado
20 de nov.
aws
Oficial

managing-data-asset-access

Grants, requests, audits and removes access to data assets in an AWS data catalog (the Amazon SageMaker Catalog supported by Amazon DataZone) through catalog subscriptions over published listings. Covers requesting a subscription, approving or rejecting requests with a recorded decision, sharing a listing with a project, revoking one or many subscriptions, and confirming grant settlement. Triggers on: give another team access to a table, request access to a dataset, approve or reject an access r

Instalações / 1h
4
GitHub Stars
3,5 mil
Atualizado
20 de nov.
aws
Oficial

managing-data-catalog

Writes and curates business metadata in an AWS data catalog: the Amazon SageMaker Catalog (Amazon DataZone), or the AWS Glue Data Catalog. Detects which catalog platform an account uses before writing. Triggers on: add a description to a table, document a dataset, write a README, set an owner, add column comments, tag with a glossary term, create a business glossary, create a metadata form, publish an asset, create a data product, run the data source crawl, delete a catalog asset, attach a conve

Instalações / 1h
4
GitHub Stars
3,5 mil
Atualizado
20 de nov.
aws
Oficial

aws-billing-and-cost-management

Analyze AWS costs, find savings, manage budgets, evaluate Savings Plans and Reserved Instances, right-size EC2/Lambda/RDS/EBS with Compute Optimizer, look up service pricing, query CUR with Athena, detect cost anomalies, scope costs to billing views, and monitor Free Tier usage. Triggers on: AWS bill, cost analysis, reduce spend, savings plan, reserved instance, right-size, budget alert, cost optimization, pricing, free tier, cost anomaly, CUR, cost audit, billing view, billing view ARN.

Instalações / 1h
2
GitHub Stars
3,5 mil
Atualizado
19 de nov.