giuseppe-trisciuoglio/developer-kit

chunking-strategy

Provides chunking strategies for RAG systems.

Voir la source

Mode d’emploi

  1. Copiez la commande et examinez le paquet avant de l’installer dans votre environnement agent.
  2. Consultez le document original pour les outils, identifiants et environnements requis.
  3. Décrivez les entrées et le livrable souhaité ; comparez la sortie aux sources.
npx skills add https://github.com/giuseppe-trisciuoglio/developer-kit --skill chunking-strategy

Avant de commencer

  • L’installation n’inclut ni abonnement au modèle, ni quota API, ni accès aux fichiers.
  • Étoiles et installations reflètent l’activité, pas un test indépendant de qualité ou de sécurité.
Document Skill original

Rendu depuis le dépôt source en conservant titres, exemples, code, tableaux, liens et images.

Chunking Strategy for RAG Systems

Overview

Provides chunking strategies for RAG systems, vector databases, and document processing. Recommends chunk sizes, overlap percentages, and boundary detection methods; validates semantic coherence; evaluates retrieval metrics.

When to Use

Use when building or optimizing RAG systems, vector search pipelines, document chunking workflows, or performance-tuning existing systems with poor retrieval quality.

Instructions

Choose Chunking Strategy

Select based on document type and use case:

  1. Fixed-Size Chunking (Level 1)
  • Use for simple documents without clear structure
  • Start with 512 tokens and 10-20% overlap
  • Adjust: 256 for factoid queries, 1024 for analytical
  1. Recursive Character Chunking (Level 2)
  • Use for documents with structural boundaries
  • Hierarchical separators: paragraphs → sentences → words
  • Customize for document types (HTML, Markdown, JSON)
  1. Structure-Aware Chunking (Level 3)
  • Use for structured content (Markdown, code, tables, PDFs)
  • Preserve semantic units: functions, sections, table blocks
  • Validate structure preservation post-split
  1. Semantic Chunking (Level 4)
  • Use for complex documents with thematic shifts
  • Embedding-based boundary detection with 0.8 similarity threshold
  • Buffer size: 3-5 sentences
  1. Advanced Methods (Level 5)
  • Late Chunking for long-context models
  • Contextual Retrieval for high-precision requirements
  • Monitor computational cost vs. retrieval gain

Reference: references/strategies.md.

Implement Chunking Pipeline

  1. Pre-process documents
  • Analyze structure, content types, information density
  • Identify multi-modal content (tables, images, code)
  1. Select parameters
  • Chunk size: embedding model context window / 4
  • Overlap: 10-20% for most cases
  • Strategy-specific settings
  1. Process and validate
  • Apply chunking strategy
  • Validate coherence: run evaluate_chunks.py --coherence (see below)
  • Test with representative documents
  1. Evaluate and iterate
  • Measure precision and recall
  • If precision < 0.7: reduce chunk_size by 25% and re-evaluate
  • If recall < 0.6: increase overlap by 10% and re-evaluate
  • Monitor latency and memory usage

Reference: references/implementation.md.

Validate Chunk Quality

Run validation commands to assess chunk quality:

bash
# Check semantic coherence (requires sentence-transformers)
python -c "
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
chunks = [...]  # your chunks
embeddings = model.encode(chunks)
similarity = (embeddings @ embeddings.T).mean()
print(f'Cohesion: {similarity:.3f}')  # target: 0.3-0.7
"

# Measure retrieval precision
python -c "
relevant = sum(1 for c in retrieved if c in relevant_chunks)
precision = relevant / len(retrieved)
print(f'Precision: {precision:.2f}')  # target: >= 0.7
"

# Check chunk size distribution
python -c "
import numpy as np
sizes = [len(c.split()) for c in chunks]
print(f'Mean: {np.mean(sizes):.0f}, Std: {np.std(sizes):.0f}')
print(f'Min: {min(sizes)}, Max: {max(sizes)}')
"

Reference: references/evaluation.md.

Examples

Fixed-Size Chunking

python
from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=256,
    chunk_overlap=25,
    length_function=len
)
chunks = splitter.split_documents(documents)

Structure-Aware Code Chunking

python
import ast

def chunk_python_code(code):
    tree = ast.parse(code)
    chunks = []
    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.ClassDef)):
            chunks.append(ast.get_source_segment(code, node))
    return chunks

Semantic Chunking

python
def semantic_chunk(text, similarity_threshold=0.8):
    sentences = split_into_sentences(text)
    embeddings = generate_embeddings(sentences)
    chunks, current = [], [sentences[0]]
    for i in range(1, len(sentences)):
        sim = cosine_similarity(embeddings[i-1], embeddings[i])
        if sim < similarity_threshold:
            chunks.append(" ".join(current))
            current = [sentences[i]]
        else:
            current.append(sentences[i])
    chunks.append(" ".join(current))
    return chunks

Best Practices

Core Principles

  • Balance context preservation with retrieval precision
  • Maintain semantic coherence within chunks
  • Optimize for embedding model context window constraints

Implementation

  • Start with fixed-size (512 tokens, 15% overlap)
  • Iterate based on document characteristics
  • Test with domain-specific documents before deployment

Pitfalls to Avoid

  • Over-chunking: context-poor small chunks
  • Under-chunking: missing information in oversized chunks
  • Ignoring semantic boundaries and document structure
  • One-size-fits-all for diverse content types

Constraints and Warnings

Resource Considerations

  • Semantic methods require significant compute resources
  • Late chunking needs long-context embedding models
  • Complex strategies increase processing latency
  • Monitor memory for large document batches

Quality Requirements

  • Validate semantic coherence post-processing
  • Test with representative documents before deployment
  • Ensure chunks maintain standalone meaning
  • Implement error handling for malformed content

References

Tâches connexes

du même dépôt

Autres Skills

Tous les Skills
giuseppe-trisciuoglio
Communauté

drizzle-orm-patterns

Provides comprehensive Drizzle ORM patterns for schema definition, CRUD operations, relations, queries, transactions, and migrations. Proactively use for any Drizzle ORM development including defining database schemas, writing type-safe queries, implementing relations, managing transactions, and setting up migrations with Drizzle Kit. Supports PostgreSQL, MySQL, SQLite, MSSQL, and CockroachDB.

Installations / 1h
3
GitHub Stars
372
Mis à jour
10 sept.
giuseppe-trisciuoglio
Communauté

tailwind-css-patterns

Provides comprehensive Tailwind CSS utility-first styling patterns including responsive design, layout utilities, flexbox, grid, spacing, typography, colors, and modern CSS best practices. Use when styling React/Vue/Svelte components, building responsive layouts, implementing design systems, or optimizing CSS workflow.

Installations / 1h
1
GitHub Stars
372
Mis à jour
10 sept.
giuseppe-trisciuoglio
Communauté

dynamodb-toolbox-patterns

Provides TypeScript patterns for DynamoDB-Toolbox v2 including schema/table/entity modeling, .build() command workflow, query/scan access patterns, batch and transaction operations, and single-table design with computed keys. Use when implementing type-safe DynamoDB access layers with DynamoDB-Toolbox v2 in TypeScript services or serverless applications.

Installations / 1h
1
GitHub Stars
371
Mis à jour
10 sept.
giuseppe-trisciuoglio
Communauté

spring-boot-dependency-injection

Provides dependency injection patterns for Spring Boot projects, including constructor-first design, optional collaborator handling, bean selection, and wiring validation. Use when creating services and configurations, replacing field injection, or troubleshooting ambiguous or fragile Spring wiring.

Installations / 1h
1
GitHub Stars
371
Mis à jour
10 sept.