nemo-curator

사용 방법

설치 명령을 복사하고 자신의 에이전트 환경에 설치하기 전에 소스 패키지를 확인하세요.
필요한 도구, 인증 정보, 지원 환경은 원본 문서에서 확인하세요.
입력과 원하는 산출물을 설명하고 출처와 결과를 대조하세요.

npx skills add https://github.com/nousresearch/hermes-agent --skill nemo-curator

시작 전 확인

설치에는 모델 구독, API 한도, 파일 접근 권한이 포함되지 않습니다.
별점 수와 설치 수는 활동 지표이며 독립적인 품질·보안 검증이 아닙니다.

원본 Skill 문서

원본 저장소의 제목, 예시, 코드, 표, 링크, 이미지를 유지해 표시합니다.

NeMo Curator - GPU-Accelerated Data Curation

NVIDIA's toolkit for preparing high-quality training data for LLMs.

When to use NeMo Curator

Use NeMo Curator when:

Preparing LLM training data from web scrapes (Common Crawl)
Need fast deduplication (16× faster than CPU)
Curating multi-modal datasets (text, images, video, audio)
Filtering low-quality or toxic content
Scaling data processing across GPU cluster

Performance:

16× faster fuzzy deduplication (8TB RedPajama v2)
40% lower TCO vs CPU alternatives
Near-linear scaling across GPU nodes

Use alternatives instead:

datatrove: CPU-based, open-source data processing
dolma: Allen AI's data toolkit
Ray Data: General ML data processing (no curation focus)

Quick start

Installation

bash

# NeMo Curator 1.x installs with uv. Extras use hyphens (PyPI-normalized):
#   text-cuda12 / text-cpu (and image/video/audio/math variants), or `all`.

# Text curation (CUDA 12)
uv pip install "nemo-curator[text-cuda12]"

# All modalities
uv pip install "nemo-curator[all]"

# CPU-only text (slower)
uv pip install "nemo-curator[text-cpu]"

Basic text curation pipeline

Major version rewrite (1.x): NeMo Curator was rewritten around a Ray-based pipeline/stage architecture. The old DocumentDataset + nemo_curator.modules.* / ScoreFilter / Modify call-the-object-on-a-dataset API from 0.x is gone. In 1.x you compose ProcessingStages into a Pipeline and run it with an executor. The exact stage/import surface differs per modality — treat the examples in this skill below as conceptual (0.x-style) and follow the current quickstart and text guide for the exact 1.x APIs rather than copying imports verbatim.

Shape of a 1.x pipeline (from the upstream quickstart):

python

from nemo_curator.pipeline import Pipeline
from nemo_curator.stages.base import ProcessingStage
from nemo_curator.stages.resources import Resources
from nemo_curator.backends.xenna import XennaExecutor
from nemo_curator.core.client import RayClient

# 1. Define/compose stages (load -> filter -> dedupe -> classify -> write).
#    Each stage declares its own Resources (CPU cores, GPU memory, replicas).
pipeline = Pipeline(name="curation", stages=[...])

# 2. Run it with an executor (Ray-backed).
client = RayClient()
client.start()
pipeline.run(XennaExecutor())
client.stop()

The 0.x-style snippets in the sections that follow illustrate the concepts (quality filtering, exact/fuzzy/semantic dedup, PII redaction, classifier filtering). For runnable 1.x code, map each concept onto the corresponding stage from the modality guide.

Data curation pipeline

Stage 1: Quality filtering

python

from nemo_curator.filters import (
    WordCountFilter,
    RepeatedLinesFilter,
    UrlRatioFilter,
    NonAlphaNumericFilter
)

# Apply 30+ heuristic filters
from nemo_curator import ScoreFilter

# Word count filter
dataset = dataset.filter(WordCountFilter(min_words=50, max_words=100000))

# Remove repetitive content
dataset = dataset.filter(RepeatedLinesFilter(max_repeated_line_fraction=0.3))

# URL ratio filter
dataset = dataset.filter(UrlRatioFilter(max_url_ratio=0.2))

Stage 2: Deduplication

Exact deduplication:

python

from nemo_curator.modules import ExactDuplicates

# Remove exact duplicates
deduped = ExactDuplicates(id_field="id", text_field="text")(dataset)

Fuzzy deduplication (16× faster on GPU):

python

from nemo_curator.modules import FuzzyDuplicates

# MinHash + LSH deduplication
fuzzy_dedup = FuzzyDuplicates(
    id_field="id",
    text_field="text",
    num_hashes=260,      # MinHash parameters
    num_buckets=20,
    hash_method="md5"
)

deduped = fuzzy_dedup(dataset)

Semantic deduplication:

python

from nemo_curator.modules import SemanticDuplicates

# Embedding-based deduplication
semantic_dedup = SemanticDuplicates(
    id_field="id",
    text_field="text",
    embedding_model="sentence-transformers/all-MiniLM-L6-v2",
    threshold=0.8  # Cosine similarity threshold
)

deduped = semantic_dedup(dataset)

Stage 3: PII redaction

python

from nemo_curator.modules import Modify
from nemo_curator.modifiers import PIIRedactor

# Redact personally identifiable information
pii_redactor = PIIRedactor(
    supported_entities=["EMAIL_ADDRESS", "PHONE_NUMBER", "PERSON", "LOCATION"],
    anonymize_action="replace"  # or "redact"
)

redacted = Modify(pii_redactor)(dataset)

Stage 4: Classifier filtering

python

from nemo_curator.classifiers import QualityClassifier

# Quality classification
quality_clf = QualityClassifier(
    model_path="nvidia/quality-classifier-deberta",
    batch_size=256,
    device="cuda"
)

# Filter low-quality documents
high_quality = dataset.filter(lambda doc: quality_clf(doc["text"]) > 0.5)

GPU acceleration

GPU vs CPU performance

Operation	CPU (16 cores)	GPU (A100)	Speedup
Fuzzy dedup (8TB)	120 hours	7.5 hours	16×
Exact dedup (1TB)	8 hours	0.5 hours	16×
Quality filtering	2 hours	0.2 hours	10×

Multi-GPU scaling

python

from nemo_curator import get_client
import dask_cuda

# Initialize GPU cluster
client = get_client(cluster_type="gpu", n_workers=8)

# Process with 8 GPUs
deduped = FuzzyDuplicates(...)(dataset)

Multi-modal curation

Image curation

python

from nemo_curator.image import (
    AestheticFilter,
    NSFWFilter,
    CLIPEmbedder
)

# Aesthetic scoring
aesthetic_filter = AestheticFilter(threshold=5.0)
filtered_images = aesthetic_filter(image_dataset)

# NSFW detection
nsfw_filter = NSFWFilter(threshold=0.9)
safe_images = nsfw_filter(filtered_images)

# Generate CLIP embeddings
clip_embedder = CLIPEmbedder(model="openai/clip-vit-base-patch32")
image_embeddings = clip_embedder(safe_images)

Video curation

python

from nemo_curator.video import (
    SceneDetector,
    ClipExtractor,
    InternVideo2Embedder
)

# Detect scenes
scene_detector = SceneDetector(threshold=27.0)
scenes = scene_detector(video_dataset)

# Extract clips
clip_extractor = ClipExtractor(min_duration=2.0, max_duration=10.0)
clips = clip_extractor(scenes)

# Generate embeddings
video_embedder = InternVideo2Embedder()
video_embeddings = video_embedder(clips)

Audio curation

python

from nemo_curator.audio import (
    ASRInference,
    WERFilter,
    DurationFilter
)

# ASR transcription
asr = ASRInference(model="nvidia/stt_en_fastconformer_hybrid_large_pc")
transcribed = asr(audio_dataset)

# Filter by WER (word error rate)
wer_filter = WERFilter(max_wer=0.3)
high_quality_audio = wer_filter(transcribed)

# Duration filtering
duration_filter = DurationFilter(min_duration=1.0, max_duration=30.0)
filtered_audio = duration_filter(high_quality_audio)

Common patterns

Web scrape curation (Common Crawl)

python

from nemo_curator import ScoreFilter, Modify
from nemo_curator.filters import *
from nemo_curator.modules import *
from nemo_curator.datasets import DocumentDataset

# Load Common Crawl data
dataset = DocumentDataset.read_parquet("common_crawl/*.parquet")

# Pipeline
pipeline = [
    # 1. Quality filtering
    WordCountFilter(min_words=100, max_words=50000),
    RepeatedLinesFilter(max_repeated_line_fraction=0.2),
    SymbolToWordRatioFilter(max_symbol_to_word_ratio=0.3),
    UrlRatioFilter(max_url_ratio=0.3),

    # 2. Language filtering
    LanguageIdentificationFilter(target_languages=["en"]),

    # 3. Deduplication
    ExactDuplicates(id_field="id", text_field="text"),
    FuzzyDuplicates(id_field="id", text_field="text", num_hashes=260),

    # 4. PII redaction
    PIIRedactor(),

    # 5. NSFW filtering
    NSFWClassifier(threshold=0.8)
]

# Execute
for stage in pipeline:
    dataset = stage(dataset)

# Save
dataset.to_parquet("curated_common_crawl/")

Distributed processing

python

from nemo_curator import get_client
from dask_cuda import LocalCUDACluster

# Multi-GPU cluster
cluster = LocalCUDACluster(n_workers=8)
client = get_client(cluster=cluster)

# Process large dataset
dataset = DocumentDataset.read_parquet("s3://large_dataset/*.parquet")
deduped = FuzzyDuplicates(...)(dataset)

# Cleanup
client.close()
cluster.close()

Performance benchmarks

Fuzzy deduplication (8TB RedPajama v2)

CPU (256 cores): 120 hours
GPU (8× A100): 7.5 hours
Speedup: 16×

Exact deduplication (1TB)

CPU (64 cores): 8 hours
GPU (4× A100): 0.5 hours
Speedup: 16×

Quality filtering (100GB)

CPU (32 cores): 2 hours
GPU (2× A100): 0.2 hours
Speedup: 10×

Cost comparison

CPU-based curation (AWS c5.18xlarge × 10):

Cost: $3.60/hour × 10 = $36/hour
Time for 8TB: 120 hours
Total: $4,320

GPU-based curation (AWS p4d.24xlarge × 2):

Cost: $32.77/hour × 2 = $65.54/hour
Time for 8TB: 7.5 hours
Total: $491.55

Savings: 89% reduction ($3,828 saved)

Supported data formats

Input: Parquet, JSONL, CSV
Output: Parquet (recommended), JSONL
WebDataset: TAR archives for multi-modal

Use cases

Production deployments:

NVIDIA used NeMo Curator to prepare Nemotron-4 training data
Open-source datasets curated: RedPajama v2, The Pile

References

[Filtering Guide](references/filtering.md) - 30+ quality filters, heuristics
[Deduplication Guide](references/deduplication.md) - Exact, fuzzy, semantic methods

Resources

GitHub: https://github.com/NVIDIA-NeMo/Curator
Docs: https://docs.nvidia.com/nemo/curator/latest/
Version: 1.2.0 (1.x is a Ray-based pipeline rewrite — see the quickstart before copying 0.x snippets)
License: Apache 2.0

사용 방법

시작 전 확인

NeMo Curator - GPU-Accelerated Data Curation

When to use NeMo Curator

Quick start

Installation

Basic text curation pipeline

Data curation pipeline

Stage 1: Quality filtering

Stage 2: Deduplication

Stage 3: PII redaction

Stage 4: Classifier filtering

GPU acceleration

GPU vs CPU performance

Multi-GPU scaling

Multi-modal curation

Image curation

Video curation

Audio curation

Common patterns

Web scrape curation (Common Crawl)

Distributed processing

Performance benchmarks

Fuzzy deduplication (8TB RedPajama v2)

Exact deduplication (1TB)

Quality filtering (100GB)

Cost comparison

Supported data formats

Use cases

References

Resources

더 많은 Skills

pixel-art

inspecting-hermes-desktop-dom

github

hermes-agent