magnus919/agent-skills

data-engineering

Design and operate data infrastructure — database operations (vector, relational, graph, time-series), ETL/ELT pipeline design (dbt patterns, incremental loading), SQL analytical patterns, data quality monitoring, schema migration, and storage infrastructur…

Vedi sorgente

Come usare

  1. Copia il comando e controlla il pacchetto prima di installarlo nel tuo ambiente agente.
  2. Controlla strumenti, credenziali e ambienti richiesti nel documento originale.
  3. Descrivi input e risultato desiderato; confronta l’output con le fonti.
npx skills add https://github.com/magnus919/agent-skills --skill data-engineering

Prima di iniziare

  • L’installazione non include abbonamenti, quote API o accesso ai file.
  • Stelle e installazioni indicano attività, non un test indipendente di qualità o sicurezza.
Documento Skill originale

Contenuto dal repository con titoli, esempi, codice, tabelle, link e immagini preservati.

Data Engineering Methodology

Data engineering is the operational backbone of data-driven systems. This methodology covers running, maintaining, and evolving data infrastructure — from relational databases and vector stores to graph databases, time-series stores, and the transformation pipelines that move data between them.

When a model proposes data transformations, load the AI boundary workflow and use the companion record.

The Data Engineer's Domain

You ownYou don't own
Database operations — schema management, indexing, backup/recovery, migration across relational, vector, graph, and time-series storesData modeling and schema design — that's the data architect
Data transformation pipelines — dbt models, ETL/ELT patterns, incremental loading, incremental strategiesStatistical analysis and experiments — that's the data scientist
Analytical SQL — window functions, CTEs, query optimization, execution plan analysis, star schema queriesTraining infrastructure and model deployment — that's the ML engineer
Graph database operations — Neo4j data modeling, Cypher queries, graph algorithms, import/exportApplication-level data access patterns — that's the developer
Time-series database operations — InfluxDB schema design, downsampling, retention policies, TelegrafInfrastructure provisioning — that's the platform engineer
Data quality monitoring — integrity checks, deduplication, anomaly detection, freshness validationVisual dashboard design — that's the analyst / product-design-and-ux
Storage infrastructure — capacity planning, performance tuning, archival strategies

Valkey and Redis decisions

For in-memory event pipelines, vector/search indexes, compatibility and sink reconciliation, read the focused guide. For open-source-oriented choices, prefer Valkey's BSD licensing and community governance; preserve explicit engine requirements and verify version-specific compatibility. Redis's AGPL option does not erase licensing or trust preferences, and this guidance makes no claim about business-community preference.

Reference Files

ReferenceWhen to load
references/sql-analytical-patterns.mdWriting analytical SQL — window functions, CTEs, execution plan reading, star schema queries, engine-specific optimization (PostgreSQL, DuckDB, ClickHouse, BigQuery, Snowflake)
references/dbt-patterns.mdDesigning data transformation pipelines with dbt — project structure, modeling layers (staging/intermediate/facts/dimensions), materializations, tests, snapshots, Jinja macros, CI/CD, dbt Mesh
references/etl-pipeline-design.mdBuilding reliable data pipelines — extraction strategies (full, incremental, CDC), transformation layers, validation gates, error handling, idempotency
references/data-quality.mdMonitoring data integrity — quality dimensions, validation rule types, anomaly detection, deduplication strategies, pipeline health signals
references/graph-databases.mdWorking with graph databases — Neo4j data modeling, Cypher query patterns (traversal, aggregation, pathfinding), import strategies, graph algorithms, pipeline integration
references/time-series-databases.mdWorking with time-series databases — InfluxDB data model (measurements, tags, fields), schema design (cardinality), downsampling, retention, Telegraf ingest, comparison with TimescaleDB/QuestDB/Prometheus
references/vector-db-operations.mdManaging vector databases — Milvus, Qdrant, Chroma — index types, collection lifecycle, dimension migrations, backup strategies
references/database-migrations.mdSchema evolution — zero-downtime migration patterns, rollback planning, versioned schemas, test-first migrations
references/backup-and-recovery.mdBackup strategies per data store type, RPO/RTO planning, WAL archiving, snapshot management, recovery plan template
references/ai-transformation-boundaries.mdDefining acceptance, retry, cost, and publication boundaries for model-assisted transformations

Related Skills

  • postgres — operating a PostgreSQL server itself: configuration review, index and query-plan diagnosis, vacuum/bloat management, WAL archiving and point-in-time recovery, replication and failover, upgrades. This skill owns the engine-specific runbooks; data-engineering owns the engine-neutral methodology.
  • supabase — Supabase platform operations: migrations, RLS, Auth, Storage, Functions, and self-hosting. To measure an agent's Supabase task competence, use its agent evals harness reference.

Core Principles

Data without integrity is noise — No pipeline, model, or dashboard is worth more than the quality of the data feeding it. Validate at every boundary.

Design for operability — Every database, pipeline, and store needs monitoring, backup, and recovery procedures defined before it goes to production. If you can't detect failure, you can't recover from it.

Idempotency is a requirement — Every pipeline should produce the same result whether it runs once or twice. Duplicate handling is not optional.

Schema changes are code changes — Every migration needs review, testing, and a rollback plan. Schema drift is technical debt with compounding interest.

Know your storage characteristics — Access patterns, retention requirements, growth rates, and consistency guarantees determine the right storage architecture. Choose based on data, not familiarity.

dallo stesso repository

Altri Skills

Tutti gli Skills
magnus919
Community

ai-governance

Design and operate AI governance: principles, decision rights, risk tiers, lifecycle gates, fairness, transparency, privacy, security, compliance mapping, maturity and board reporting across SaaS, API, self-hosted and agentic systems. Use to admit, update or retire Agent Skills, MCP servers and A2A capabilities; approve an agent for production actions; or decide earned-autonomy promotion, reduction or revocation. Cover GxP, ALCOA+, data integrity, electronic records, assurance and QMS interfaces

Installazioni / 1h
1
GitHub Stars
172
Aggiornato
8 nov
magnus919
Community

amazon-memo

Create, adapt, or revise Amazon-style narrative decision memos, six-pagers, PR/FAQs, and short decision briefs, including reading meetings and async review. Use for choosing the right memo form, evidence-led reasoning, and revising mechanical AI prose for human readers. Do not use for general copywriting, document rendering, or replacing incident reports and technical design records.

Installazioni / 1h
1
GitHub Stars
172
Aggiornato
8 nov
magnus919
Community

analog-occultism

Use this skill to create subject-neutral images and videos in an analog occultism / industrial CRT noir aesthetic: near-monochrome archival technical atmosphere, severe low-key lighting, tactile signal degradation, industrial geometry, and quiet unresolved tension. Use for portraits, products, environments, interfaces, diagrams, abstract forms, stills, and motion. Do not use for colorful cyberpunk, vaporwave, glossy advertising, conventional horror, or clean digital renders.

Installazioni / 1h
1
GitHub Stars
172
Aggiornato
8 nov
magnus919
Community

binary-analysis

Plan static inspection of unknown binary files with a deterministic CLI whose Ghidra operations are currently unimplemented and fail closed. Use when you need to inspect a PE, ELF, or Mach-O file — triage suspicious binaries, map imported APIs, decompile functions, trace call paths, or produce structured evidence reports. Do not use for runtime analysis (debugging, dynamic tracing, sandbox execution), for modifying or patching binaries, or for binaries you already know everything about. The skil

Installazioni / 1h
1
GitHub Stars
172
Aggiornato
8 nov