giuseppe-trisciuoglio/developer-kit

chunking-strategy

Provides chunking strategies for RAG systems.

소스 보기

사용 방법

  1. 설치 명령을 복사하고 자신의 에이전트 환경에 설치하기 전에 소스 패키지를 확인하세요.
  2. 필요한 도구, 인증 정보, 지원 환경은 원본 문서에서 확인하세요.
  3. 입력과 원하는 산출물을 설명하고 출처와 결과를 대조하세요.
npx skills add https://github.com/giuseppe-trisciuoglio/developer-kit --skill chunking-strategy

시작 전 확인

  • 설치에는 모델 구독, API 한도, 파일 접근 권한이 포함되지 않습니다.
  • 별점 수와 설치 수는 활동 지표이며 독립적인 품질·보안 검증이 아닙니다.
원본 Skill 문서

원본 저장소의 제목, 예시, 코드, 표, 링크, 이미지를 유지해 표시합니다.

Chunking Strategy for RAG Systems

Overview

Provides chunking strategies for RAG systems, vector databases, and document processing. Recommends chunk sizes, overlap percentages, and boundary detection methods; validates semantic coherence; evaluates retrieval metrics.

When to Use

Use when building or optimizing RAG systems, vector search pipelines, document chunking workflows, or performance-tuning existing systems with poor retrieval quality.

Instructions

Choose Chunking Strategy

Select based on document type and use case:

  1. Fixed-Size Chunking (Level 1)
  • Use for simple documents without clear structure
  • Start with 512 tokens and 10-20% overlap
  • Adjust: 256 for factoid queries, 1024 for analytical
  1. Recursive Character Chunking (Level 2)
  • Use for documents with structural boundaries
  • Hierarchical separators: paragraphs → sentences → words
  • Customize for document types (HTML, Markdown, JSON)
  1. Structure-Aware Chunking (Level 3)
  • Use for structured content (Markdown, code, tables, PDFs)
  • Preserve semantic units: functions, sections, table blocks
  • Validate structure preservation post-split
  1. Semantic Chunking (Level 4)
  • Use for complex documents with thematic shifts
  • Embedding-based boundary detection with 0.8 similarity threshold
  • Buffer size: 3-5 sentences
  1. Advanced Methods (Level 5)
  • Late Chunking for long-context models
  • Contextual Retrieval for high-precision requirements
  • Monitor computational cost vs. retrieval gain

Reference: references/strategies.md.

Implement Chunking Pipeline

  1. Pre-process documents
  • Analyze structure, content types, information density
  • Identify multi-modal content (tables, images, code)
  1. Select parameters
  • Chunk size: embedding model context window / 4
  • Overlap: 10-20% for most cases
  • Strategy-specific settings
  1. Process and validate
  • Apply chunking strategy
  • Validate coherence: run evaluate_chunks.py --coherence (see below)
  • Test with representative documents
  1. Evaluate and iterate
  • Measure precision and recall
  • If precision < 0.7: reduce chunk_size by 25% and re-evaluate
  • If recall < 0.6: increase overlap by 10% and re-evaluate
  • Monitor latency and memory usage

Reference: references/implementation.md.

Validate Chunk Quality

Run validation commands to assess chunk quality:

bash
# Check semantic coherence (requires sentence-transformers)
python -c "
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
chunks = [...]  # your chunks
embeddings = model.encode(chunks)
similarity = (embeddings @ embeddings.T).mean()
print(f'Cohesion: {similarity:.3f}')  # target: 0.3-0.7
"

# Measure retrieval precision
python -c "
relevant = sum(1 for c in retrieved if c in relevant_chunks)
precision = relevant / len(retrieved)
print(f'Precision: {precision:.2f}')  # target: >= 0.7
"

# Check chunk size distribution
python -c "
import numpy as np
sizes = [len(c.split()) for c in chunks]
print(f'Mean: {np.mean(sizes):.0f}, Std: {np.std(sizes):.0f}')
print(f'Min: {min(sizes)}, Max: {max(sizes)}')
"

Reference: references/evaluation.md.

Examples

Fixed-Size Chunking

python
from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=256,
    chunk_overlap=25,
    length_function=len
)
chunks = splitter.split_documents(documents)

Structure-Aware Code Chunking

python
import ast

def chunk_python_code(code):
    tree = ast.parse(code)
    chunks = []
    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.ClassDef)):
            chunks.append(ast.get_source_segment(code, node))
    return chunks

Semantic Chunking

python
def semantic_chunk(text, similarity_threshold=0.8):
    sentences = split_into_sentences(text)
    embeddings = generate_embeddings(sentences)
    chunks, current = [], [sentences[0]]
    for i in range(1, len(sentences)):
        sim = cosine_similarity(embeddings[i-1], embeddings[i])
        if sim < similarity_threshold:
            chunks.append(" ".join(current))
            current = [sentences[i]]
        else:
            current.append(sentences[i])
    chunks.append(" ".join(current))
    return chunks

Best Practices

Core Principles

  • Balance context preservation with retrieval precision
  • Maintain semantic coherence within chunks
  • Optimize for embedding model context window constraints

Implementation

  • Start with fixed-size (512 tokens, 15% overlap)
  • Iterate based on document characteristics
  • Test with domain-specific documents before deployment

Pitfalls to Avoid

  • Over-chunking: context-poor small chunks
  • Under-chunking: missing information in oversized chunks
  • Ignoring semantic boundaries and document structure
  • One-size-fits-all for diverse content types

Constraints and Warnings

Resource Considerations

  • Semantic methods require significant compute resources
  • Late chunking needs long-context embedding models
  • Complex strategies increase processing latency
  • Monitor memory for large document batches

Quality Requirements

  • Validate semantic coherence post-processing
  • Test with representative documents before deployment
  • Ensure chunks maintain standalone meaning
  • Implement error handling for malformed content

References

관련 작업

같은 저장소의 Skills

더 많은 Skills

모든 Skills
giuseppe-trisciuoglio
커뮤니티

drizzle-orm-patterns

Provides comprehensive Drizzle ORM patterns for schema definition, CRUD operations, relations, queries, transactions, and migrations. Proactively use for any Drizzle ORM development including defining database schemas, writing type-safe queries, implementing relations, managing transactions, and setting up migrations with Drizzle Kit. Supports PostgreSQL, MySQL, SQLite, MSSQL, and CockroachDB.

1시간 설치 수
3
GitHub Stars
372
업데이트
9월 10일
giuseppe-trisciuoglio
커뮤니티

tailwind-css-patterns

Provides comprehensive Tailwind CSS utility-first styling patterns including responsive design, layout utilities, flexbox, grid, spacing, typography, colors, and modern CSS best practices. Use when styling React/Vue/Svelte components, building responsive layouts, implementing design systems, or optimizing CSS workflow.

1시간 설치 수
1
GitHub Stars
372
업데이트
9월 10일
giuseppe-trisciuoglio
커뮤니티

dynamodb-toolbox-patterns

Provides TypeScript patterns for DynamoDB-Toolbox v2 including schema/table/entity modeling, .build() command workflow, query/scan access patterns, batch and transaction operations, and single-table design with computed keys. Use when implementing type-safe DynamoDB access layers with DynamoDB-Toolbox v2 in TypeScript services or serverless applications.

1시간 설치 수
1
GitHub Stars
371
업데이트
9월 10일
giuseppe-trisciuoglio
커뮤니티

spring-boot-dependency-injection

Provides dependency injection patterns for Spring Boot projects, including constructor-first design, optional collaborator handling, bean selection, and wiring validation. Use when creating services and configurations, replacing field injection, or troubleshooting ambiguous or fragile Spring wiring.

1시간 설치 수
1
GitHub Stars
371
업데이트
9월 10일