wshobson/agents

ml-pipeline-workflow

Build end-to-end MLOps pipelines from data preparation through model training, validation, and production deployment.

소스 보기

사용 방법

  1. 설치 명령을 복사하고 자신의 에이전트 환경에 설치하기 전에 소스 패키지를 확인하세요.
  2. 필요한 도구, 인증 정보, 지원 환경은 원본 문서에서 확인하세요.
  3. 입력과 원하는 산출물을 설명하고 출처와 결과를 대조하세요.
npx skills add https://github.com/wshobson/agents --skill ml-pipeline-workflow

시작 전 확인

  • 설치에는 모델 구독, API 한도, 파일 접근 권한이 포함되지 않습니다.
  • 별점 수와 설치 수는 활동 지표이며 독립적인 품질·보안 검증이 아닙니다.
원본 Skill 문서

원본 저장소의 제목, 예시, 코드, 표, 링크, 이미지를 유지해 표시합니다.

ML Pipeline Workflow

Complete end-to-end MLOps pipeline orchestration from data preparation through model deployment.

Overview

This skill provides comprehensive guidance for building production ML pipelines that handle the full lifecycle: data ingestion → preparation → training → validation → deployment → monitoring.

When to Use This Skill

  • Building new ML pipelines from scratch
  • Designing workflow orchestration for ML systems
  • Implementing data → model → deployment automation
  • Setting up reproducible training workflows
  • Creating DAG-based ML orchestration
  • Integrating ML components into production systems

What This Skill Provides

Core Capabilities

  1. Pipeline Architecture
  • End-to-end workflow design
  • DAG orchestration patterns (Airflow, Dagster, Kubeflow)
  • Component dependencies and data flow
  • Error handling and retry strategies
  1. Data Preparation
  • Data validation and quality checks
  • Feature engineering pipelines
  • Data versioning and lineage
  • Train/validation/test splitting strategies
  1. Model Training
  • Training job orchestration
  • Hyperparameter management
  • Experiment tracking integration
  • Distributed training patterns
  1. Model Validation
  • Validation frameworks and metrics
  • A/B testing infrastructure
  • Performance regression detection
  • Model comparison workflows
  1. Deployment Automation
  • Model serving patterns
  • Canary deployments
  • Blue-green deployment strategies
  • Rollback mechanisms

Reference Documentation

See the references/ directory for detailed guides:

  • data-preparation.md - Data cleaning, validation, and feature engineering
  • model-training.md - Training workflows and best practices
  • model-validation.md - Validation strategies and metrics
  • model-deployment.md - Deployment patterns and serving architectures

Assets and Templates

The assets/ directory contains:

  • pipeline-dag.yaml.template - DAG template for workflow orchestration
  • training-config.yaml - Training configuration template
  • validation-checklist.md - Pre-deployment validation checklist

Usage Patterns

Basic Pipeline Setup

python
# 1. Define pipeline stages
stages = [
    "data_ingestion",
    "data_validation",
    "feature_engineering",
    "model_training",
    "model_validation",
    "model_deployment"
]

# 2. Configure dependencies
# See assets/pipeline-dag.yaml.template for full example

Production Workflow

  1. Data Preparation Phase
  • Ingest raw data from sources
  • Run data quality checks
  • Apply feature transformations
  • Version processed datasets
  1. Training Phase
  • Load versioned training data
  • Execute training jobs
  • Track experiments and metrics
  • Save trained models
  1. Validation Phase
  • Run validation test suite
  • Compare against baseline
  • Generate performance reports
  • Approve for deployment
  1. Deployment Phase
  • Package model artifacts
  • Deploy to serving infrastructure
  • Configure monitoring
  • Validate production traffic

Best Practices

Pipeline Design

  • Modularity: Each stage should be independently testable
  • Idempotency: Re-running stages should be safe
  • Observability: Log metrics at every stage
  • Versioning: Track data, code, and model versions
  • Failure Handling: Implement retry logic and alerting

Data Management

  • Use data validation libraries (Great Expectations, TFX)
  • Version datasets with DVC or similar tools
  • Document feature engineering transformations
  • Maintain data lineage tracking

Model Operations

  • Separate training and serving infrastructure
  • Use model registries (MLflow, Weights & Biases)
  • Implement gradual rollouts for new models
  • Monitor model performance drift
  • Maintain rollback capabilities

Deployment Strategies

  • Start with shadow deployments
  • Use canary releases for validation
  • Implement A/B testing infrastructure
  • Set up automated rollback triggers
  • Monitor latency and throughput

Integration Points

Orchestration Tools

  • Apache Airflow: DAG-based workflow orchestration
  • Dagster: Asset-based pipeline orchestration
  • Kubeflow Pipelines: Kubernetes-native ML workflows
  • Prefect: Modern dataflow automation

Experiment Tracking

  • MLflow for experiment tracking and model registry
  • Weights & Biases for visualization and collaboration
  • TensorBoard for training metrics

Deployment Platforms

  • AWS SageMaker for managed ML infrastructure
  • Google Vertex AI for GCP deployments
  • Azure ML for Azure cloud
  • OCI Data Science for Oracle Cloud Infrastructure deployments
  • Kubernetes + KServe for cloud-agnostic serving

Progressive Disclosure

Start with the basics and gradually add complexity:

  1. Level 1: Simple linear pipeline (data → train → deploy)
  2. Level 2: Add validation and monitoring stages
  3. Level 3: Implement hyperparameter tuning
  4. Level 4: Add A/B testing and gradual rollouts
  5. Level 5: Multi-model pipelines with ensemble strategies

Common Patterns

Batch Training Pipeline

yaml
# See assets/pipeline-dag.yaml.template
stages:
  - name: data_preparation
    dependencies: []
  - name: model_training
    dependencies: [data_preparation]
  - name: model_evaluation
    dependencies: [model_training]
  - name: model_deployment
    dependencies: [model_evaluation]

Real-time Feature Pipeline

python
# Stream processing for real-time features
# Combined with batch training
# See references/data-preparation.md

Continuous Training

python
# Automated retraining on schedule
# Triggered by data drift detection
# See references/model-training.md

Troubleshooting

Common Issues

  • Pipeline failures: Check dependencies and data availability
  • Training instability: Review hyperparameters and data quality
  • Deployment issues: Validate model artifacts and serving config
  • Performance degradation: Monitor data drift and model metrics

Debugging Steps

  1. Check pipeline logs for each stage
  2. Validate input/output data at boundaries
  3. Test components in isolation
  4. Review experiment tracking metrics
  5. Inspect model artifacts and metadata

Next Steps

After setting up your pipeline:

  1. Explore hyperparameter-tuning skill for optimization
  2. Learn experiment-tracking-setup for MLflow/W&B
  3. Review model-deployment-patterns for serving strategies
  4. Implement monitoring with observability tools

Related Skills

  • experiment-tracking-setup: MLflow and Weights & Biases integration
  • hyperparameter-tuning: Automated hyperparameter optimization
  • model-deployment-patterns: Advanced deployment strategies

관련 작업

같은 저장소의 Skills

더 많은 Skills

모든 Skills
wshobson
커뮤니티

typescript-advanced-types

Master TypeScript's advanced type system including generics, conditional types, mapped types, template literals, and utility types for building type-safe applications. Use when implementing complex type logic, creating reusable type utilities, or ensuring compile-time type safety in TypeScript projects.

누적 설치 수
10.1만
GitHub Stars
4.3만
업데이트
11월 21일
wshobson
커뮤니티

tailwind-design-system

Build scalable design systems with Tailwind CSS v4, design tokens, component libraries, and responsive patterns. Use when creating component libraries, implementing design systems, or standardizing UI patterns.

누적 설치 수
7.5만
GitHub Stars
4.3만
업데이트
11월 18일
wshobson
커뮤니티

kpi-dashboard-design

Design effective KPI dashboards with metrics selection, visualization best practices, and real-time monitoring patterns. Use this skill when building an executive SaaS metrics dashboard tracking MRR, churn, and LTV/CAC ratios; designing an operations center with live service health and request throughput; creating a cohort retention analysis view for a product team; or debugging a dashboard where metrics contradict each other due to inconsistent calculation methodology.

1시간 설치 수
2
GitHub Stars
4.3만
업데이트
11월 18일
wshobson
커뮤니티

market-sizing-analysis

Calculate TAM/SAM/SOM for market opportunities using top-down, bottom-up, and value theory methodologies. Use this skill when sizing markets, estimating addressable revenue, validating market opportunity for a new venture, or building investor-ready market analysis for a startup pitch or business plan.

1시간 설치 수
2
GitHub Stars
4.3만
업데이트
11월 18일