magnus919/agent-skills

data-engineering

Design and operate data infrastructure — database operations (vector, relational, graph, time-series), ETL/ELT pipeline design (dbt patterns, incremental loading), SQL analytical patterns, data quality monitoring, schema migration, and storage infrastructur…

ソースを見る

使い方

  1. インストールコマンドをコピーし、自分の環境へ導入する前にソースを確認します。
  2. 必要なツール、認証情報、対応環境を原文で確認します。
  3. 入力と成果物を指定し、出力を資料と照合します。
npx skills add https://github.com/magnus919/agent-skills --skill data-engineering

開始前の確認

  • 導入にはモデル契約、API枠、ファイルへのアクセス許可は含まれません。
  • スター数や導入数は活動の指標であり、独立した品質・安全性試験ではありません。
リポジトリの原文

見出し、例、コード、表、リンク、参照画像を含む原文を表示しています。

Data Engineering Methodology

Data engineering is the operational backbone of data-driven systems. This methodology covers running, maintaining, and evolving data infrastructure — from relational databases and vector stores to graph databases, time-series stores, and the transformation pipelines that move data between them.

When a model proposes data transformations, load the AI boundary workflow and use the companion record.

The Data Engineer's Domain

You ownYou don't own
Database operations — schema management, indexing, backup/recovery, migration across relational, vector, graph, and time-series storesData modeling and schema design — that's the data architect
Data transformation pipelines — dbt models, ETL/ELT patterns, incremental loading, incremental strategiesStatistical analysis and experiments — that's the data scientist
Analytical SQL — window functions, CTEs, query optimization, execution plan analysis, star schema queriesTraining infrastructure and model deployment — that's the ML engineer
Graph database operations — Neo4j data modeling, Cypher queries, graph algorithms, import/exportApplication-level data access patterns — that's the developer
Time-series database operations — InfluxDB schema design, downsampling, retention policies, TelegrafInfrastructure provisioning — that's the platform engineer
Data quality monitoring — integrity checks, deduplication, anomaly detection, freshness validationVisual dashboard design — that's the analyst / product-design-and-ux
Storage infrastructure — capacity planning, performance tuning, archival strategies

Valkey and Redis decisions

For in-memory event pipelines, vector/search indexes, compatibility and sink reconciliation, read the focused guide. For open-source-oriented choices, prefer Valkey's BSD licensing and community governance; preserve explicit engine requirements and verify version-specific compatibility. Redis's AGPL option does not erase licensing or trust preferences, and this guidance makes no claim about business-community preference.

Reference Files

ReferenceWhen to load
references/sql-analytical-patterns.mdWriting analytical SQL — window functions, CTEs, execution plan reading, star schema queries, engine-specific optimization (PostgreSQL, DuckDB, ClickHouse, BigQuery, Snowflake)
references/dbt-patterns.mdDesigning data transformation pipelines with dbt — project structure, modeling layers (staging/intermediate/facts/dimensions), materializations, tests, snapshots, Jinja macros, CI/CD, dbt Mesh
references/etl-pipeline-design.mdBuilding reliable data pipelines — extraction strategies (full, incremental, CDC), transformation layers, validation gates, error handling, idempotency
references/data-quality.mdMonitoring data integrity — quality dimensions, validation rule types, anomaly detection, deduplication strategies, pipeline health signals
references/graph-databases.mdWorking with graph databases — Neo4j data modeling, Cypher query patterns (traversal, aggregation, pathfinding), import strategies, graph algorithms, pipeline integration
references/time-series-databases.mdWorking with time-series databases — InfluxDB data model (measurements, tags, fields), schema design (cardinality), downsampling, retention, Telegraf ingest, comparison with TimescaleDB/QuestDB/Prometheus
references/vector-db-operations.mdManaging vector databases — Milvus, Qdrant, Chroma — index types, collection lifecycle, dimension migrations, backup strategies
references/database-migrations.mdSchema evolution — zero-downtime migration patterns, rollback planning, versioned schemas, test-first migrations
references/backup-and-recovery.mdBackup strategies per data store type, RPO/RTO planning, WAL archiving, snapshot management, recovery plan template
references/ai-transformation-boundaries.mdDefining acceptance, retry, cost, and publication boundaries for model-assisted transformations

Related Skills

  • postgres — operating a PostgreSQL server itself: configuration review, index and query-plan diagnosis, vacuum/bloat management, WAL archiving and point-in-time recovery, replication and failover, upgrades. This skill owns the engine-specific runbooks; data-engineering owns the engine-neutral methodology.
  • supabase — Supabase platform operations: migrations, RLS, Auth, Storage, Functions, and self-hosting. To measure an agent's Supabase task competence, use its agent evals harness reference.

Core Principles

Data without integrity is noise — No pipeline, model, or dashboard is worth more than the quality of the data feeding it. Validate at every boundary.

Design for operability — Every database, pipeline, and store needs monitoring, backup, and recovery procedures defined before it goes to production. If you can't detect failure, you can't recover from it.

Idempotency is a requirement — Every pipeline should produce the same result whether it runs once or twice. Duplicate handling is not optional.

Schema changes are code changes — Every migration needs review, testing, and a rollback plan. Schema drift is technical debt with compounding interest.

Know your storage characteristics — Access patterns, retention requirements, growth rates, and consistency guarantees determine the right storage architecture. Choose based on data, not familiarity.

同じリポジトリから

関連する Skills

すべての Skills
magnus919
コミュニティ

ai-governance

Design and operate AI governance: principles, decision rights, risk tiers, lifecycle gates, fairness, transparency, privacy, security, compliance mapping, maturity and board reporting across SaaS, API, self-hosted and agentic systems. Use to admit, update or retire Agent Skills, MCP servers and A2A capabilities; approve an agent for production actions; or decide earned-autonomy promotion, reduction or revocation. Cover GxP, ALCOA+, data integrity, electronic records, assurance and QMS interfaces

1時間の導入数
1
GitHub Stars
172
更新日
11月8日
magnus919
コミュニティ

amazon-memo

Create, adapt, or revise Amazon-style narrative decision memos, six-pagers, PR/FAQs, and short decision briefs, including reading meetings and async review. Use for choosing the right memo form, evidence-led reasoning, and revising mechanical AI prose for human readers. Do not use for general copywriting, document rendering, or replacing incident reports and technical design records.

1時間の導入数
1
GitHub Stars
172
更新日
11月8日
magnus919
コミュニティ

analog-occultism

Use this skill to create subject-neutral images and videos in an analog occultism / industrial CRT noir aesthetic: near-monochrome archival technical atmosphere, severe low-key lighting, tactile signal degradation, industrial geometry, and quiet unresolved tension. Use for portraits, products, environments, interfaces, diagrams, abstract forms, stills, and motion. Do not use for colorful cyberpunk, vaporwave, glossy advertising, conventional horror, or clean digital renders.

1時間の導入数
1
GitHub Stars
172
更新日
11月8日
magnus919
コミュニティ

binary-analysis

Plan static inspection of unknown binary files with a deterministic CLI whose Ghidra operations are currently unimplemented and fail closed. Use when you need to inspect a PE, ELF, or Mach-O file — triage suspicious binaries, map imported APIs, decompile functions, trace call paths, or produce structured evidence reports. Do not use for runtime analysis (debugging, dynamic tracing, sandbox execution), for modifying or patching binaries, or for binaries you already know everything about. The skil

1時間の導入数
1
GitHub Stars
172
更新日
11月8日