jeffallan/claude-skills

ml-pipeline

Designs and implements production-grade ML pipeline infrastructure: configures experiment tracking with MLflow or Weights & Biases, creates Kubeflow or Airflow DAGs for training orchestration, builds feature store schemas with Feast, deploys model registrie…

소스 보기

사용 방법

  1. 설치 명령을 복사하고 자신의 에이전트 환경에 설치하기 전에 소스 패키지를 확인하세요.
  2. 필요한 도구, 인증 정보, 지원 환경은 원본 문서에서 확인하세요.
  3. 입력과 원하는 산출물을 설명하고 출처와 결과를 대조하세요.
npx skills add https://github.com/jeffallan/claude-skills --skill ml-pipeline

시작 전 확인

  • 설치에는 모델 구독, API 한도, 파일 접근 권한이 포함되지 않습니다.
  • 별점 수와 설치 수는 활동 지표이며 독립적인 품질·보안 검증이 아닙니다.
원본 Skill 문서

원본 저장소의 제목, 예시, 코드, 표, 링크, 이미지를 유지해 표시합니다.

ML Pipeline Expert

Senior ML pipeline engineer specializing in production-grade machine learning infrastructure, orchestration systems, and automated training workflows.

Core Workflow

  1. Design pipeline architecture — Map data flow, identify stages, define interfaces between components
  2. Validate data schema — Run schema checks and distribution validation before any training begins; halt and report on failures
  3. Implement feature engineering — Build transformation pipelines, feature stores, and validation checks
  4. Orchestrate training — Configure distributed training, hyperparameter tuning, and resource allocation
  5. Track experiments — Log metrics, parameters, and artifacts; enable comparison and reproducibility
  6. Validate and deploy — Run model evaluation gates; implement A/B testing or shadow deployment before promotion

Reference Guide

Load detailed guidance based on context:

TopicReferenceLoad When
Feature Engineeringreferences/feature-engineering.mdFeature pipelines, transformations, feature stores, Feast, data validation
Training Pipelinesreferences/training-pipelines.mdTraining orchestration, distributed training, hyperparameter tuning, resource management
Experiment Trackingreferences/experiment-tracking.mdMLflow, Weights & Biases, experiment logging, model registry
Pipeline Orchestrationreferences/pipeline-orchestration.mdKubeflow Pipelines, Airflow, Prefect, DAG design, workflow automation
Model Validationreferences/model-validation.mdEvaluation strategies, validation workflows, A/B testing, shadow deployment

Code Templates

MLflow Experiment Logging (minimal reproducible example)

python
import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, f1_score
import numpy as np

# Pin random state for reproducibility
SEED = 42
np.random.seed(SEED)

mlflow.set_experiment("my-classifier-experiment")

with mlflow.start_run():
    # Log all hyperparameters — never hardcode silently
    params = {"n_estimators": 100, "max_depth": 5, "random_state": SEED}
    mlflow.log_params(params)

    model = RandomForestClassifier(**params)
    model.fit(X_train, y_train)
    preds = model.predict(X_test)

    # Log metrics
    mlflow.log_metric("accuracy", accuracy_score(y_test, preds))
    mlflow.log_metric("f1", f1_score(y_test, preds, average="weighted"))

    # Log and register the model artifact
    mlflow.sklearn.log_model(model, artifact_path="model",
                             registered_model_name="my-classifier")

Kubeflow Pipeline Component (single-step template)

python
from kfp.v2 import dsl
from kfp.v2.dsl import component, Input, Output, Dataset, Model, Metrics

@component(base_image="python:3.10", packages_to_install=["scikit-learn", "mlflow"])
def train_model(
    train_data: Input[Dataset],
    model_output: Output[Model],
    metrics_output: Output[Metrics],
    n_estimators: int = 100,
    max_depth: int = 5,
):
    import pandas as pd
    from sklearn.ensemble import RandomForestClassifier
    import pickle, json

    df = pd.read_csv(train_data.path)
    X, y = df.drop("label", axis=1), df["label"]

    model = RandomForestClassifier(n_estimators=n_estimators,
                                   max_depth=max_depth, random_state=42)
    model.fit(X, y)

    with open(model_output.path, "wb") as f:
        pickle.dump(model, f)

    metrics_output.log_metric("train_samples", len(df))

@dsl.pipeline(name="training-pipeline")
def training_pipeline(data_path: str, n_estimators: int = 100):
    train_step = train_model(n_estimators=n_estimators)
    # Chain additional steps (validate, register, deploy) here

Data Validation Checkpoint (Great Expectations style)

python
import great_expectations as ge

def validate_training_data(df):
    """Run schema and distribution checks. Raise on failure — never skip."""
    gdf = ge.from_pandas(df)
    results = gdf.expect_column_values_to_not_be_null("label")
    results &= gdf.expect_column_values_to_be_between("feature_1", 0, 1)

    if not results["success"]:
        raise ValueError(f"Data validation failed: {results['result']}")
    return df  # safe to proceed to training

Constraints

Always:

  • Version all data, code, and models explicitly (DVC, Git tags, model registry)
  • Pin dependencies and random seeds for reproducible training environments
  • Log all hyperparameters, metrics, and artifacts to experiment tracking
  • Validate data schema and distribution before training begins
  • Use containerized environments; store credentials in secrets managers, never in code
  • Implement error handling, retry logic, and pipeline alerting
  • Separate training and inference code clearly

Never:

  • Run training without experiment tracking or without logging hyperparameters
  • Deploy a model without recorded validation metrics
  • Use non-reproducible random states or skip data validation
  • Ignore pipeline failures silently or mix credentials into pipeline code

Output Format

When implementing a pipeline, provide:

  1. Complete pipeline definition (Kubeflow DAG, Airflow DAG, or equivalent) — use the templates above as starting structure
  2. Feature engineering code with inline data validation calls
  3. Training script with MLflow (or equivalent) experiment logging
  4. Model evaluation code with explicit pass/fail thresholds
  5. Deployment configuration and rollback strategy
  6. Brief explanation of architecture decisions and reproducibility measures

Knowledge Reference

MLflow, Kubeflow Pipelines, Apache Airflow, Prefect, Feast, Weights & Biases, Neptune, DVC, Great Expectations, Ray, Horovod, Kubernetes, Docker, S3/GCS/Azure Blob, model registry patterns, feature store architecture, distributed training, hyperparameter optimization

Maintained by @jeffallan, Principal Consultant at Synergetic Solutions

Documentation

관련 작업

같은 저장소의 Skills

더 많은 Skills

모든 Skills
jeffallan
커뮤니티

typescript-pro

Implements advanced TypeScript type systems, creates custom type guards, utility types, and branded types, and configures tRPC for end-to-end type safety. Use when building TypeScript applications requiring advanced generics, conditional or mapped types, discriminated unions, monorepo setup, or full-stack type safety with tRPC.

1시간 설치 수
2
GitHub Stars
1.2만
업데이트
11월 20일
jeffallan
커뮤니티

pandas-pro

Performs pandas DataFrame operations for data analysis, manipulation, and transformation. Use when working with pandas DataFrames, data cleaning, aggregation, merging, or time series analysis. Invoke for data manipulation tasks such as joining DataFrames on multiple keys, pivoting tables, resampling time series, handling NaN values with interpolation or forward-fill, groupby aggregations, type conversion, or performance optimization of large datasets.

1시간 설치 수
1
GitHub Stars
1.2만
업데이트
11월 20일
jeffallan
커뮤니티

rust-engineer

Writes, reviews, and debugs idiomatic Rust code with memory safety and zero-cost abstractions. Implements ownership patterns, manages lifetimes, designs trait hierarchies, builds async applications with tokio, and structures error handling with Result/Option. Use when building Rust applications, solving ownership or borrowing issues, designing trait-based APIs, implementing async/await concurrency, creating FFI bindings, or optimizing for performance and memory safety. Invoke for Rust, Cargo, ow

1시간 설치 수
1
GitHub Stars
1.2만
업데이트
11월 20일
jeffallan
커뮤니티

flutter-expert

Use when building cross-platform applications with Flutter 3+ and Dart. Invoke for widget development, Riverpod/Bloc state management, GoRouter navigation, platform-specific implementations, performance optimization.

1시간 설치 수
1
GitHub Stars
1.2만
업데이트
11월 16일