aws/agent-toolkit-for-aws

managing-amazon-kinesis-data-streams

Operates data delivery and queues in Amazon Kinesis Data Streams (KDS).

Quelltext ansehen

Verwendung

  1. Installationsbefehl kopieren und das Quellpaket vor der Installation in der eigenen Agentenumgebung prüfen.
  2. Benötigte Werkzeuge, Zugangsdaten und Hosts im Original prüfen.
  3. Eingabe und gewünschtes Ergebnis beschreiben und die Ausgabe mit den Quellen abgleichen.
npx skills add https://github.com/aws/agent-toolkit-for-aws --skill managing-amazon-kinesis-data-streams

Vor dem Start

  • Die Installation umfasst weder Modellabo noch API-Kontingent oder Dateizugriffsrechte.
  • Sterne und Installationen zeigen Aktivität, keine unabhängige Qualitäts- oder Sicherheitsprüfung.
Originales Skill-Dokument

Aus dem Quell-Repository gerendert; Überschriften, Beispiele, Code, Tabellen, Links und Bilder bleiben erhalten.

Amazon Kinesis Data Streams

Overview

Domain expertise for two Amazon Kinesis Data Streams features:

  • Streaming tables and Amazon S3 delivery (data delivery) - serverless, fully managed delivery from a KDS stream to Apache Iceberg tables on S3 Tables or to general-purpose S3 buckets. No consumers, Firehose, or Flink required for append-only delivery. Delivery is configured as a channel on a stream with CreateChannel. Channels consume no shard capacity and no enhanced fan-out slots.
  • Queues - a queue delivers records from an on-demand stream to a queue consumer and tracks the delivery state of each record. Consumer instances receive records, process them, and acknowledge them, with no shards, leases, or checkpoints to manage. A queue has no retention period of its own: a record that is never acknowledged, including a dead-letter record, expires when the stream's retention period ends for that record.

The two features are independent. A stream can have channels and queues at the same time, alongside existing stream consumers.

The AWS MCP server is recommended for sandboxed execution and audit logging of AWS operations. Without it, run the same operations with the AWS CLI or an AWS SDK.

Which Workflow Do You Need?

If a Kinesis request is ambiguous, ask: "Are you delivering stream data to S3 or Iceberg, or consuming records in your own application?"

Ask the same when a user mentions a dead-letter queue. The two features handle failed records in unrelated ways: a channel's dead-letter queue is an S3 bucket, while a queue marks a record that keeps failing as a dead-letter record and stops returning it from ReceiveQueueRecords.

Data delivery: streaming tables and S3

Customer IntentReference
Deliver stream data to Apache Iceberg tables on S3 Tables - setup, IAM, Glue schema, type mapping, partitioning, channel lifecyclereferences/data-delivery-streaming-tables.md
Deliver stream data to a general-purpose S3 bucket as objects - setup, IAM, compression, storage class, output key templates, channel lifecyclereferences/data-delivery-general-purpose-delivery-to-s3.md
Build a lakehouse / data lake from Kinesis; make streaming data queryable in Athenareferences/data-delivery-streaming-tables.md
Zero-ops, serverless alternative to Firehose or Kinesis consumers for KDS to S3 deliveryreferences/data-delivery-general-purpose-delivery-to-s3.md
CloudWatch metrics and alarms, delivery logging, CloudTrail, encryption, channel states, quotas, naming rules, what can be updatedreferences/data-delivery-monitoring-security-and-limits.md
Channel is ACTIVE but no data arrives; records landing in the channel's DLQ; rising data freshnessStart with references/data-delivery-monitoring-security-and-limits.md, then the destination-specific troubleshooting table
Deliver to both S3 and S3 Tables from one streamSupported - create one channel per destination

Queues in Kinesis Data Streams

Customer IntentReference
What queues are; whether a workload fits (ordering, filtering, partition keys, delivery guarantees); create a queue, including on a stream that IaC manages; build a queue consumer; visibility timeout; redelivery and other troubleshootingreferences/queues-getting-started.md
Migrate a KCL, GetRecords, or Lambda consumer to a queue consumerreferences/queues-migration.md (plan template: assets/queues-migration-plan-template.md)
Dead-letter records; MaxReceiveCount; inspect, restore, or acknowledge themreferences/queues-dead-letter-records.md
IAM policies for queue consumers and operators; security considerations; operation behavior; quotas; queue CloudWatch metricsreferences/queues-operations-and-iam.md

Prerequisites

Data delivery and queues both require an ON_DEMAND stream. Provisioned-mode streams are not supported. Use UpdateStreamMode to switch an existing stream; switching modes doesn't disrupt producers or consumers. Don't recreate the stream, which would lose the records already in it.

  • Data delivery also needs an IAM service execution role trusted by kinesis.amazonaws.com, and a destination in the same Region as the stream.
  • Queues need an AWS CLI or AWS SDK version that has the queue operations, so check before running or generating any queue calls. For the CLI, run aws kinesis receive-queue-records help. If the command isn't found, ask the user before updating, then follow AWS CLI update instructions. For an SDK, update the Kinesis client to its latest release with the language's package manager. Queues also need their own IAM actions. A policy that lists only stream actions does not grant them. CreateQueue also requires kinesis:AssociateStreamsWithQueue on the stream ARN.

Hard Limits to Check First

Data delivery

These rule out delivery entirely, so confirm them before designing a solution:

  • Append-only - no CDC, upserts, or deletes
  • No schema evolution - a schema change means deleting and recreating the channel
  • No backfill - only records produced after the channel is ACTIVE are delivered
  • No transformations - records are delivered as-is
  • New table per channel - cannot deliver into an existing Iceberg table
  • Data freshness is 300-900 seconds - not sub-minute
  • Same Region only - cross-account is supported, cross-Region is not

If any of these block the customer, recommend Managed Service for Apache Flink instead.

Queues

Queues fit stateless processing, where each record can be processed on its own, without reference to other records, for example microservice integration, order processing and fulfillment, notifications, background jobs, and per-record inference. Before building or migrating a queue consumer, ask whether the workload needs any of these:

  • Strict ordering per partition key or shard. A queue doesn't necessarily deliver records in the order producers wrote them.
  • Partition-key grouping - the same consumer handling all records for a key. A queue distributes records across consumers independently of shards.
  • Exactly-once processing - processing a record more than once would cause a problem, such as a duplicate write. A queue delivers each record at least once, so processing must be idempotent.

If the whole workload needs strict ordering or partition-key grouping (stateful processing, such as live analytics, personalization, or anomaly and pattern detection), read the stream directly with shared throughput or enhanced fan-out, for example with KCL. If only part does, offer a hybrid. If duplicates are the only problem, the processing must be made idempotent first.

Also check the producer: a queue does not de-aggregate records written with KPL aggregation. See references/queues-getting-started.md.

Guardrail - where this skill's own files live (MCP vs local install)

This skill can be loaded two ways, and they resolve the skill's own bundled files from different places. Determine how the skill was loaded before reading a reference or asset:

  • Loaded through the AWS MCP `retrieve_skill` tool: The skill is not

installed on the local filesystem. You MUST fetch each reference or asset via retrieve_skill with the file parameter (e.g. file="references/queues-getting-started.md" or file="assets/queues-migration-plan-template.md"). Do NOT file_read these paths from the local or working directory, and do NOT search the filesystem for them - they are not there, and any local file that happens to match the name is unrelated to this skill.

  • Installed locally (e.g. .kiro/skills/managing-amazon-kinesis-data-streams/ or

~/.claude/skills/managing-amazon-kinesis-data-streams/): Read files from the local skill directory using relative paths.

This distinction applies only to the skill's own packaged files. User data and session artifacts are always read from and written to the user's working directory. Never fetch or write customer data through retrieve_skill.

Additional Resources

aus demselben Repository

Weitere Skills

Alle Skills
aws
Offiziell

deploying-containers-to-amazon-lightsail

Deploys, hosts, or runs a container or Docker app on Amazon Lightsail Container Service and returns a live, managed HTTPS URL at a flat monthly price — no cluster, VPC, load balancer, or registry to set up. Used when deploying a container image to Lightsail from a local Docker build, a public registry image, or a GitHub repo, or when a Lightsail container deployment fails or won't serve — including when the request doesn't name Lightsail but asks to run one always-on container behind a public UR

Installationen / 1h
4
GitHub Stars
3534
Aktualisiert
20. Nov.
aws
Offiziell

managing-data-asset-access

Grants, requests, audits and removes access to data assets in an AWS data catalog (the Amazon SageMaker Catalog supported by Amazon DataZone) through catalog subscriptions over published listings. Covers requesting a subscription, approving or rejecting requests with a recorded decision, sharing a listing with a project, revoking one or many subscriptions, and confirming grant settlement. Triggers on: give another team access to a table, request access to a dataset, approve or reject an access r

Installationen / 1h
4
GitHub Stars
3534
Aktualisiert
20. Nov.
aws
Offiziell

managing-data-catalog

Writes and curates business metadata in an AWS data catalog: the Amazon SageMaker Catalog (Amazon DataZone), or the AWS Glue Data Catalog. Detects which catalog platform an account uses before writing. Triggers on: add a description to a table, document a dataset, write a README, set an owner, add column comments, tag with a glossary term, create a business glossary, create a metadata form, publish an asset, create a data product, run the data source crawl, delete a catalog asset, attach a conve

Installationen / 1h
4
GitHub Stars
3534
Aktualisiert
20. Nov.
aws
Offiziell

aws-billing-and-cost-management

Analyze AWS costs, find savings, manage budgets, evaluate Savings Plans and Reserved Instances, right-size EC2/Lambda/RDS/EBS with Compute Optimizer, look up service pricing, query CUR with Athena, detect cost anomalies, scope costs to billing views, and monitor Free Tier usage. Triggers on: AWS bill, cost analysis, reduce spend, savings plan, reserved instance, right-size, budget alert, cost optimization, pricing, free tier, cost anomaly, CUR, cost audit, billing view, billing view ARN.

Installationen / 1h
2
GitHub Stars
3527
Aktualisiert
19. Nov.