# Chunking (information retrieval)

> Source: https://aiwiki.ai/wiki/chunking
> Updated: 2026-06-07
> Categories: Information Retrieval, Natural Language Processing
> From AI Wiki (https://aiwiki.ai), a free encyclopedia of artificial intelligence. Quote with attribution.

**Chunking** is the preprocessing step in document indexing and [retrieval-augmented generation](/wiki/retrieval_augmented_generation_rag) (RAG) systems in which a long document or corpus is split into smaller passages, called chunks, that are then embedded and stored in a [vector database](/wiki/vector_database) for retrieval. Chunking exists because [embedding models](/wiki/embeddings) and large language models impose finite input lengths: most [Sentence-BERT](/wiki/sentence-bert) variants accept up to 512 subword tokens, the default `all-MiniLM-L6-v2` model truncates to 256 tokens, and even long-context encoders such as OpenAI's `text-embedding-3-large` cap input at 8,192 tokens.[^1][^2][^3] Chunking decisions, namely chunk size, overlap, and boundary policy, materially affect retrieval recall and downstream answer faithfulness; small chunks concentrate the relevant span but lose surrounding context, whereas large chunks preserve context but dilute the embedding signal across irrelevant material.[^4][^5] Common strategies range from fixed-size character or token splits, through recursive splitters that try paragraph, sentence, and word boundaries in turn, to embedding-driven semantic chunking and the more recent contextual and late chunking methods introduced by Anthropic, Jina AI, and Voyage AI in 2024 and 2025.[^6][^7][^8]

## Background and motivation

Dense retrieval pipelines map text into fixed-dimensional vectors so that semantic similarity can be approximated by [cosine similarity](/wiki/cosine_similarity) or dot product between query and document vectors.[^4] Two engineering constraints make chunking unavoidable. First, the [transformer](/wiki/transformer) encoders behind dense retrievers have a maximum sequence length, beyond which tokens are silently truncated. The `all-MiniLM-L6-v2` model, one of the most widely used [Sentence-BERT](/wiki/sentence-bert) checkpoints, truncates inputs longer than 256 subword tokens by default and was trained on sequences capped at 128 tokens, so it does not generalise well to longer passages even if the underlying BERT configuration allows 512 positions.[^2] OpenAI's third-generation embedding models, `text-embedding-3-small` and `text-embedding-3-large`, both accept up to 8,192 tokens and produce up to 1,536 and 3,072 dimensional vectors respectively, but inputs longer than that must be either truncated or split.[^3][^9] Second, even when an embedding model has a long context window, the LLM that consumes the retrieved chunks at generation time has its own [context window](/wiki/context_window) limit, and packing too few, overly long chunks into the prompt wastes the [reranker](/wiki/reranker) and generator on irrelevant text.[^4]

Beyond hard limits, chunking serves a quality function: a single embedding compresses an entire passage into one vector, so the longer the passage, the more semantic detail collapses. When a 1,000-token chunk discusses three different sub-topics, the resulting vector points toward the centroid of those topics and is unlikely to closely match any query about a single one of them. Smaller chunks therefore raise retrieval precision because the relevant span dominates the chunk's vector, but they also raise the risk that the answer-bearing information was placed in a neighbouring chunk and missed.[^5][^10] Pinecone's chunking guide frames the tradeoff explicitly: smaller chunks capture granular semantics, larger chunks retain broader context, and the right size depends on the document type, the embedding model's training regime, and the expected query complexity.[^4]

## Fixed-size chunking

Fixed-size chunking, the simplest strategy, splits text every N characters or tokens, optionally with overlap between adjacent chunks. Greg Kamradt's "5 Levels of Text Splitting" tutorial, an influential reference among RAG practitioners, calls fixed-size character splitting "Level 1" and describes it as a useful starting point for understanding the chunking problem but not a method he recommends for production systems.[^7][^11] Amazon Bedrock Knowledge Bases exposes fixed-size chunking as a first-class option, allowing users to configure the maximum number of tokens per chunk and a percentage overlap between consecutive chunks.[^12] Bedrock's "default chunking" mode, by contrast, targets approximately 300 tokens per chunk while honoring sentence boundaries, so the resulting chunks are not strictly equal in length but never cut a sentence in half.[^12]

Fixed-size approaches have two well-known failure modes. First, they cut across sentences, tables, and code structures: a 512-token window starting mid-paragraph may begin in the middle of one sentence and end in the middle of another, leaving the embedding model to encode a fragment that lacks subject or predicate. Second, the size parameter is hard to choose blind. Pinecone's guide and many community guides converge on a default range of 200 to 512 tokens with 10 to 20 percent overlap, but NVIDIA's evaluation on FinanceBench with 1,024-token chunks found that 15 percent overlap outperformed 10 and 20 percent, and Chroma's evaluation framework shows that the OpenAI Assistants default of 800 tokens with 400-token overlap had below-average recall on a multi-corpus benchmark.[^4][^13][^14]

### Chunk overlap

Chunk overlap is the practice of repeating a tail of one chunk at the head of the next, so that information located near a chunk boundary appears in two chunks rather than being split. The standard heuristic in the industry is 10 to 20 percent of chunk size; for a 1,000-character chunk, this means 100 to 200 characters of overlap.[^13] The purpose is to mitigate the worst-case scenario in which the answer to a query straddles a boundary and would otherwise appear at the end of one chunk and the start of the next, hurting the embedding similarity of both. Overlap is not a free lunch: it inflates the number of stored vectors and the cost of retrieval, and beyond roughly 30 percent the marginal gain in recall is small relative to the storage and latency penalty.[^13]

## Recursive chunking

The most widely adopted general-purpose chunker is LangChain's `RecursiveCharacterTextSplitter`, which Greg Kamradt has called "the swiss army knife of splitters" and his usual first choice when prototyping.[^7] The recursive splitter accepts a target `chunk_size` and an ordered list of separators, by default `["\n\n", "\n", " ", ""]`, representing paragraph breaks, line breaks, spaces, and empty strings.[^15][^6] It first tries to split the input on the strongest separator (paragraph break); if the resulting segments still exceed `chunk_size`, it recurses on the next separator within each oversized segment, and so on down to the character level. The intent is to keep paragraphs together where possible and only break into smaller units when the paragraph itself exceeds the target size.[^15]

LangChain's API exposes a `chunk_overlap` parameter that adds overlap between adjacent output chunks and a `length_function` that defaults to character count but can be swapped for a token counter to match the embedding model's tokenizer.[^6][^15] For languages such as Chinese, Japanese, and Thai, which lack whitespace word boundaries, the documentation recommends supplying custom separators including punctuation and zero-width spaces so that the splitter does not break mid-word.[^15] Internal tests at Chroma showed that with default parameters, `RecursiveCharacterTextSplitter` performed "relatively poorly" on synthetic query evaluation, which the authors attribute to the gap between default character-based size and tokenizer-aware sizing rather than to a flaw in the recursive idea itself.[^14]

[LlamaIndex](/wiki/llamaindex) provides an analogous component called `SentenceSplitter`, with default `chunk_size=1024` tokens and `chunk_overlap=200` tokens, that tries to keep sentences and paragraphs together so that fewer hanging sentence fragments occur at chunk boundaries than with the older `TokenTextSplitter`.[^16]

## Sentence-aware and structure-aware chunking

A second family of strategies splits text on linguistic or structural boundaries rather than on raw character counts.

### Sentence boundary chunking

Sentence-based splitters use NLTK's `sent_tokenize` or spaCy's statistical sentence segmenter to first break a document into sentences, then group adjacent sentences until a target token budget is reached.[^17] NLTK uses a deterministic, rule-based tokenizer based on punctuation patterns; spaCy uses a trained statistical model that handles ambiguous cases such as abbreviations more robustly.[^17] LangChain wraps both in `NLTKTextSplitter` and `SpacyTextSplitter` adapters. Sentence chunking avoids mid-sentence splits, which is the main visual artefact of fixed-size methods, but it does not guarantee that the resulting chunks are semantically coherent: a single boundary placed between two related sentences can still split a coherent idea across chunks.

### Markdown- and document-structure-aware chunking

When documents have explicit hierarchical structure, the splitter can use that structure as a cue. LangChain's `MarkdownHeaderTextSplitter` accepts a configurable list of header levels, by default `("#", "Header 1"), ("##", "Header 2"), ("###", "Header 3")`, and produces one chunk per section, attaching the chain of enclosing headers to each chunk as metadata so that downstream retrieval can filter or rerank by section.[^18][^19] The splitter strips the header line from the chunk body by default, controlled by the `strip_headers` parameter, and overlap does not cross section boundaries.[^18] A common pipeline chains `MarkdownHeaderTextSplitter` with `RecursiveCharacterTextSplitter` so that very long sections are further subdivided while shorter sections remain whole.[^19] Analogous splitters exist for HTML, LaTeX, and PDF documents that exploit the corresponding structural markers.[^4]

### Code-aware chunking

Source code is poorly served by character-based chunking because functions, classes, and control structures rarely align with fixed token windows. The cAST method, published in June 2025 by researchers from Carnegie Mellon University and Augment Code, parses source files into an Abstract Syntax Tree (AST) with `tree-sitter` and then applies a recursive split-then-merge algorithm that greedily merges adjacent AST nodes while respecting a configured chunk size, recursively subdivides oversized nodes, and measures size in non-whitespace characters.[^20] cAST is designed for four properties: syntactic integrity (chunks correspond to whole syntactic units), high information density, language invariance across more than a hundred grammars supported by tree-sitter, and plug-and-play behaviour (concatenating chunks recovers the original file).[^20] In experiments across RepoEval, CrossCodeEval, and SWE-bench, cAST improved RepoEval Recall@5 by 4.3 points, SWE-bench Pass@1 by 2.67 points, and CrossCodeEval exact match by 4.3 points compared with line-based baselines, with the largest single gain (5.5 points) coming from StarCoder2 on RepoEval.[^20]

## Semantic chunking

[Semantic chunking](/wiki/semantic_chunking), introduced as Level 4 of Kamradt's framework, places chunk boundaries where the topic shifts rather than at a fixed token count.[^7] The basic algorithm first splits the document into sentences, embeds each sentence, then walks adjacent sentences computing the [cosine similarity](/wiki/cosine_similarity) between their embeddings; when the similarity between two adjacent sentence groups drops below a configurable threshold, the algorithm declares a chunk boundary.[^21] [LlamaIndex](/wiki/llamaindex) implements this idea as `SemanticSplitterNodeParser` with three primary hyperparameters: `buffer_size` (the number of surrounding sentences combined before embedding, default 1), `breakpoint_percentile_threshold` (the percentile of inter-sentence distances above which a boundary is placed, typically 80 to 95), and an `embed_model` pointer to the embedding provider used for the similarity computation.[^22] AWS Bedrock Knowledge Bases exposes a similar set of parameters: maximum tokens per chunk, buffer size of surrounding sentences, and a breakpoint percentile threshold, with higher thresholds yielding fewer and larger chunks.[^12]

Semantic chunking trades simplicity for adaptability. A document that switches frequently between topics will produce many chunks, while a long discussion of a single topic will produce a few large ones. The cost is that an embedding model must be invoked once per sentence (or per small sentence group) during ingestion, adding latency and dollars per document. Bedrock's documentation notes explicitly that semantic chunking incurs additional foundation-model cost during ingestion compared with fixed or hierarchical chunking.[^12]

Kamradt also describes a Level 5 "agentic" chunker that prompts an LLM to identify split points, treating chunking as a reasoning task rather than a similarity computation, and an "experimental" Level intended for the regime where token costs approach zero.[^7] Chroma's open evaluation framework instantiates a similar idea as `LLMChunker`, which reached the highest recall (91.9 percent) among all chunkers tested with OpenAI's `text-embedding-3-large` at five retrieved chunks, although its precision (3.9) was lower than the `ClusterSemanticChunker` baseline (8.0 precision, 91.3 percent recall) at the same 200-token target size.[^14]

## Contextual retrieval

In September 2024 [Anthropic](/wiki/anthropic) published "Introducing [Contextual Retrieval](/wiki/contextual_retrieval)," describing a method that prepends a short, document-aware context string to each chunk before embedding and BM25 indexing.[^23] The motivation is the observation that a chunk such as "The company's revenue grew by 3% over the previous quarter" loses the antecedents required to retrieve it: a query for revenue growth at a specific company cannot match this chunk on its own because the chunk does not name the company or the time period.[^23] The Anthropic method uses Claude (in the published cookbook, Claude 3 Haiku) to generate a one- or two-sentence context for each chunk given the full source document, then concatenates the context with the chunk before embedding and before constructing the [BM25](/wiki/bm25) inverted index.[^23]

Anthropic reported retrieval-failure reductions on a benchmark of codebases, fiction, ArXiv papers, and other science papers: contextual embeddings alone cut failure rates from 5.7 percent to 3.7 percent (a 35 percent reduction), contextual embeddings combined with contextual BM25 cut them to 2.9 percent (a 49 percent reduction), and adding a reranker brought them to 1.9 percent (a 67 percent reduction). Their cost model assumes 800-token chunks within 8,000-token documents, a 50-token context-generation instruction, and roughly 100 tokens of generated context per chunk, yielding a one-time ingestion cost of $1.02 per million document tokens when [prompt caching](/wiki/prompt_caching) is used to amortize the document content across all chunks of the same document.[^23] AWS published an Amazon Bedrock Knowledge Bases integration that implements the technique, and [Anthropic](/wiki/anthropic)'s public cookbook includes a reference implementation.[^24]

## Late chunking

A parallel September 2024 paper from Jina AI, "Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models" by Michael Günther, Isabelle Mohr, Daniel James Williams, Bo Wang, and Han Xiao, proposes a different solution to the same context-loss problem.[^25] Rather than rewriting chunks before embedding, late chunking changes the order of operations: it first runs the entire long document through a long-context embedding model in a single pass, producing one token-level vector per token in the document, and only then segments the resulting vector sequence into chunks and applies mean pooling within each chunk. Because every token's contextual representation already attended to every other token during the forward pass, each resulting chunk vector encodes information from the surrounding document, not only from the chunk itself.[^7][^25]

Jina's published evaluation on BeIR datasets shows consistent gains over naive chunking on the same encoder: SciFact nDCG@10 rose from 64.20 to 66.10, TRECCOVID from 63.36 to 64.70, and NFCorpus from 23.46 to 29.98, with the authors observing that "the longer the document, the more effective the late chunking strategy becomes."[^26] The technique requires no extra training and works with any long-context embedding model that uses mean pooling, with Jina's own [Jina Embeddings v3](/wiki/jina_embeddings_v3) and its predecessor v2-base-en (both with 8,192-token windows) as the reference targets.[^25][^26] Subsequent integration posts from Milvus, Elasticsearch, and other vector-search vendors describe how to wire late chunking into production retrieval stacks.

## Contextualised chunk embeddings (voyage-context-3)

In July 2025 [Voyage AI](/wiki/voyage_ai) (by then a subsidiary of MongoDB) released `voyage-context-3`, an embedding model that internalises the contextualisation step rather than performing it as a preprocessing pass.[^8][^27] The model accepts a list of chunks from the same source document, runs them through a single forward pass with cross-chunk attention, and emits one embedding per chunk that has been conditioned on the rest of the document. Voyage describe this as "contextualised chunk embeddings" and present them as a drop-in replacement for standard embedding APIs that requires no metadata augmentation by the user.[^27]

On Voyage's evaluation of 93 retrieval datasets spanning nine domains (web reviews, law, medicine, long documents, technical documentation, code, finance, conversations, and multilingual text), `voyage-context-3` at NDCG@10 reportedly outperforms `text-embedding-3-large` by 14.24 percent at chunk-level retrieval and 12.56 percent at document-level retrieval, Cohere embed v4 by 7.89 percent and 5.64 percent, Jina v3 late chunking by 23.66 percent and 6.76 percent, and Anthropic's contextual retrieval by 20.54 percent and 2.40 percent.[^8] The model supports 2048, 1024, 512, and 256 dimensional outputs and 32-bit float, signed and unsigned 8-bit integer, and binary quantisations; Voyage reports that the binary 512-dimensional variant matches `text-embedding-3-large` in retrieval quality while using roughly 0.5 percent of the storage.[^8]

## Chunk size and overlap heuristics

The chunk-size question has been studied empirically and the answers are corpus-specific, but a few generalisations recur across guides from Pinecone, NVIDIA, Chroma, and the LangChain community:

| Embedding model | Max input | Practical chunk size |
|---|---|---|
| `all-MiniLM-L6-v2` ([Sentence-BERT](/wiki/sentence-bert)) | 256 tokens (default) | 128 to 256 tokens[^2] |
| BERT-base / SBERT 512-token variants | 512 tokens | 256 to 512 tokens[^1] |
| OpenAI `text-embedding-3-large` | 8,192 tokens | 256 to 1,024 tokens (typical), up to 8,192 for full-doc[^3][^9] |
| Jina embeddings v3 | 8,192 tokens | 512 to 2,048 tokens (or full-doc with late chunking)[^26] |
| Voyage `voyage-context-3` | full-document | chunk size mostly irrelevant; sensitivity 2.06 percent across configurations[^8] |

For chunk overlap, 10 to 20 percent of chunk size is the broadly cited default; NVIDIA's FinanceBench experiment found 15 percent optimal at 1,024-token chunks, intermediate between the 10 and 20 percent endpoints.[^13] Above 30 percent overlap, returns diminish quickly while index size grows linearly.[^13]

## Evaluation

Because chunking sits upstream of every other RAG component, evaluating chunking quality requires a measurable retrieval task, not just qualitative inspection. The standard metric is Recall@K, the fraction of test queries for which at least one relevant chunk appears in the top-K results returned by the retriever.[^28] BEIR, a widely used benchmark for dense retrievers, packages 18 datasets across nine retrieval tasks and reports nDCG@10, MAP, Precision@K, and Recall@K side by side; researchers comparing chunking strategies often report results on a subset of BEIR plus their own domain corpus.[^28] [MTEB](/wiki/mteb) (the Massive Text Embedding Benchmark) plays a complementary role for embedding-model comparison.

Chroma's chunking evaluation, published in 2024, generates synthetic queries with GPT-4 Turbo against five diverse corpora totalling 328,208 tokens, filters them by embedding similarity, and computes token-level precision, recall, and Intersection over Union (IoU) to capture both whether the relevant span was retrieved and how much extraneous content the retrieved chunks contain.[^14] The framework allows direct comparison of chunkers (recursive, semantic, LLM-driven) at controlled chunk sizes with the same embedding model. Chroma's headline finding is that default parameter choices in popular libraries are often far from optimal: the OpenAI Assistants default of 800 tokens with 400-token overlap underperformed both smaller and better-tuned alternatives.[^14] Pinecone's practitioner guidance is to run a small representative test set at chunk sizes of 200, 400, and 600 tokens, measure Recall@5, and pick the configuration that maximises it for the specific corpus.[^4]

## Hierarchical and parent-child chunking

Hierarchical chunking, also called parent-child chunking, indexes two granularities at once: small "child" chunks that are precise enough for embedding similarity to discriminate, and larger "parent" chunks that contain the surrounding context.[^12] At retrieval time, the system embeds the query, finds the top-K child chunks by vector similarity, and then returns each child's parent as the actual context given to the LLM. The intent is to combine the precision of small chunks with the contextual completeness of larger ones. Amazon Bedrock Knowledge Bases supports hierarchical chunking with two levels, allowing the user to configure parent and child token sizes and an overlap, and warns that the returned number of results may be lower than requested because multiple matched children may map to the same parent.[^12]

[LlamaIndex](/wiki/llamaindex) implements a similar idea as the "auto-merging retriever," which retrieves leaf chunks and merges them into their parent when enough siblings are returned, and as the "sentence-window retriever," which embeds individual sentences but returns each retrieved sentence together with a fixed window of surrounding sentences to the generator.[^16]

## Implementations across major libraries

[LangChain](/wiki/langchain) groups its text splitters in the `langchain_text_splitters` package. The defaults include `CharacterTextSplitter` (Level 1), `RecursiveCharacterTextSplitter` (Level 2 and the recommended default for generic text), `TokenTextSplitter` and per-tokenizer variants for OpenAI and HuggingFace models, `MarkdownHeaderTextSplitter` and `HTMLHeaderTextSplitter` (Level 3 for structured documents), and integrations with NLTK and spaCy for sentence-level splitting.[^15][^18]

[LlamaIndex](/wiki/llamaindex)'s analogous abstractions are "node parsers," which produce `Node` objects (the LlamaIndex equivalent of a chunk). The library ships `SentenceSplitter` (default `chunk_size=1024`, `chunk_overlap=200`), `TokenTextSplitter`, `SentenceWindowNodeParser`, `MarkdownNodeParser`, and `SemanticSplitterNodeParser` (the embedding-walk implementation).[^16][^22]

Vector database vendors have largely converged on offering chunking inside their managed ingestion pipelines. [Pinecone](/wiki/pinecone)'s "Chunking Strategies" guide recommends fixed-size chunking as the starting point for most applications and provides examples spanning content-aware, structure-aware, semantic, and contextual chunking with LLMs.[^4] [Weaviate](/wiki/weaviate) and [Qdrant](/wiki/qdrant) both publish chunking guides and integrations with LangChain and LlamaIndex. [Anthropic](/wiki/anthropic)'s Claude Cookbook hosts a reference implementation of contextual retrieval that practitioners can adapt to their own corpora.[^24]

## Limitations and open problems

Despite a decade of dense-retrieval research, chunking remains a poorly automated step in most RAG pipelines. Three limitations recur across surveys and evaluations:

First, **chunk size is corpus- and query-dependent**, and there is no universal best size. Pinecone, NVIDIA, and Chroma all recommend empirical tuning rather than defaults, but doing so requires a labelled or synthetic evaluation set that many production teams do not have.[^4][^13][^14] AI21's 2024 work on multi-scale retrieval argues that the optimal chunk size depends on the query type and proposes indexing the same corpus at multiple granularities to dispatch each query to the appropriate scale.[^29]

Second, **chunk boundaries lose context**, which is the central problem that contextual retrieval, late chunking, and contextualised chunk embeddings all attempt to solve. Each solution carries its own cost: contextual retrieval requires an LLM pass per chunk at ingestion time (cheap with [prompt caching](/wiki/prompt_caching) but still nonzero); late chunking depends on long-context encoders that may not exist for every language or domain; contextualised chunk embeddings tie the user to a specific embedding provider.[^23][^25][^8]

Third, **evaluation infrastructure is immature**. Token-level metrics such as Chroma's IoU give a more faithful picture than document-level Recall@K, but they require synthetic-query generation pipelines that are themselves sensitive to the LLM prompt and the corpus. The same chunking method can win or lose by several percentage points depending on the evaluation dataset, the embedding model, and the chunk size considered, so single-number comparisons across providers should be treated with caution.[^14]

## See also

- [Retrieval-augmented generation (RAG)](/wiki/retrieval_augmented_generation_rag)
- [Retrieval-augmented generation](/wiki/retrieval_augmented_generation)
- [Embeddings](/wiki/embeddings)
- [Embedding vector](/wiki/embedding_vector)
- [Vector database](/wiki/vector_database)
- [Sentence-BERT (SBERT)](/wiki/sentence-bert)
- [Cosine similarity](/wiki/cosine_similarity)
- [BM25 (Okapi BM25)](/wiki/bm25)
- [Hybrid search](/wiki/hybrid_search)
- [HyDE (Hypothetical Document Embeddings)](/wiki/hyde)
- [Semantic chunking](/wiki/semantic_chunking)
- [Contextual retrieval](/wiki/contextual_retrieval)
- [Reranker](/wiki/reranker)
- [MTEB (Massive Text Embedding Benchmark)](/wiki/mteb)
- [LangChain](/wiki/langchain)
- [LlamaIndex](/wiki/llamaindex)
- [Pinecone](/wiki/pinecone)
- [Weaviate](/wiki/weaviate)
- [Qdrant](/wiki/qdrant)
- [Jina Embeddings v3](/wiki/jina_embeddings_v3)
- [Voyage AI](/wiki/voyage_ai)
- [Voyage-3](/wiki/voyage_3)
- [Anthropic](/wiki/anthropic)
- [Prompt Caching](/wiki/prompt_caching)
- [Context window](/wiki/context_window)
- [Tokenization](/wiki/tokenization)

## References

[^1]: Reimers, Nils and Gurevych, Iryna, "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", arXiv:1908.10084, 2019-08-27. https://arxiv.org/pdf/1908.10084. Accessed 2026-05-25.
[^2]: Sentence-Transformers, "sentence-transformers/all-MiniLM-L6-v2 model card", Hugging Face, 2024. https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2. Accessed 2026-05-25.
[^3]: OpenAI, "text-embedding-3-large Model", OpenAI Platform Documentation, 2024-01-25. https://platform.openai.com/docs/models/text-embedding-3-large. Accessed 2026-05-25.
[^4]: Pinecone, "Chunking Strategies for LLM Applications", Pinecone Learn, 2023. https://www.pinecone.io/learn/chunking-strategies/. Accessed 2026-05-25.
[^5]: DataStax, "Split Documents", RAGStack Documentation, 2024. https://docs.datastax.com/en/ragstack/default-architecture/splitting.html. Accessed 2026-05-25.
[^6]: LangChain, "RecursiveCharacterTextSplitter Reference", LangChain Python API Reference, 2024. https://reference.langchain.com/python/langchain-text-splitters/character/RecursiveCharacterTextSplitter. Accessed 2026-05-25.
[^7]: Kamradt, Greg, "5 Levels Of Text Splitting / Chunking", Full Stack Retrieval, 2024. https://community.fullstackretrieval.com/document-loaders/text-splitting. Accessed 2026-05-25.
[^8]: Voyage AI, "Introducing voyage-context-3: focused chunk-level details with global document context", Voyage AI Blog, 2025-07-23. https://blog.voyageai.com/2025/07/23/voyage-context-3/. Accessed 2026-05-25.
[^9]: OpenAI, "New embedding models and API updates", OpenAI Blog, 2024-01-25. https://openai.com/index/new-embedding-models-and-api-updates/. Accessed 2026-05-25.
[^10]: Firecrawl, "Best Chunking Strategies for RAG (and LLMs) in 2026", Firecrawl Blog, 2026. https://www.firecrawl.dev/blog/best-chunking-strategies-rag. Accessed 2026-05-25.
[^11]: Kamradt, Greg, "5 Levels Of Text Splitting/Chunking", X (formerly Twitter), 2023-09-06. https://x.com/GregKamradt/status/1699465826485862543. Accessed 2026-05-25.
[^12]: Amazon Web Services, "How content chunking works for knowledge bases", AWS Bedrock User Guide, 2024. https://docs.aws.amazon.com/bedrock/latest/userguide/kb-chunking.html. Accessed 2026-05-25.
[^13]: NVIDIA, "Finding the Best Chunking Strategy for Accurate AI Responses", NVIDIA Technical Blog, 2024. https://developer.nvidia.com/blog/finding-the-best-chunking-strategy-for-accurate-ai-responses/. Accessed 2026-05-25.
[^14]: Smith, Brandon and Troynikov, Anton, "Evaluating Chunking Strategies for Retrieval", Chroma Research, 2024-07-03. https://www.trychroma.com/research/evaluating-chunking. Accessed 2026-05-25.
[^15]: LangChain, "Splitting recursively - Text splitter integration guide", LangChain Documentation, 2024. https://docs.langchain.com/oss/python/integrations/splitters/recursive_text_splitter. Accessed 2026-05-25.
[^16]: LlamaIndex, "Sentence splitter", LlamaIndex Documentation, 2024. https://docs.llamaindex.ai/en/stable/api_reference/node_parsers/sentence_splitter/. Accessed 2026-05-25.
[^17]: Bombay Softwares, "RAG Pipeline: How It Transforms Natural Language Processing", Bombay Softwares Blog, 2024. https://www.bombaysoftwares.com/blog/rag-pipeline-how-it-transforms-natural-language-processing. Accessed 2026-05-25.
[^18]: LangChain, "Split markdown - text splitter integration", LangChain Documentation, 2024. https://docs.langchain.com/oss/python/integrations/splitters/markdown_header_metadata_splitter. Accessed 2026-05-25.
[^19]: LangChain, "How to split Markdown by Headers", LangChain v0.2 Documentation, 2024. https://python.langchain.com/v0.2/docs/how_to/markdown_header_metadata_splitter/. Accessed 2026-05-25.
[^20]: Zhang, Yilin, Zhao, Xinran, Wang, Zora Zhiruo, Yang, Chenyang, Wei, Jiayi and Wu, Tongshuang, "cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Abstract Syntax Tree", arXiv:2506.15655, 2025-06-18. https://arxiv.org/abs/2506.15655. Accessed 2026-05-25.
[^21]: Superlinked, "Semantic Chunking", Superlinked VectorHub, 2024. https://superlinked.com/vectorhub/articles/semantic-chunking. Accessed 2026-05-25.
[^22]: LlamaIndex, "Semantic splitter", LlamaIndex Documentation, 2024. https://docs.llamaindex.ai/en/latest/api_reference/node_parsers/semantic_splitter/. Accessed 2026-05-25.
[^23]: Anthropic, "Introducing Contextual Retrieval", Anthropic News, 2024-09-19. https://www.anthropic.com/news/contextual-retrieval. Accessed 2026-05-25.
[^24]: Anthropic, "Enhancing RAG with contextual retrieval", Claude Cookbook, 2024. https://platform.claude.com/cookbook/capabilities-contextual-embeddings-guide. Accessed 2026-05-25.
[^25]: Günther, Michael, Mohr, Isabelle, Williams, Daniel James, Wang, Bo and Xiao, Han, "Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models", arXiv:2409.04701, 2024-09-07. https://arxiv.org/abs/2409.04701. Accessed 2026-05-25.
[^26]: Günther, Michael and Xiao, Han, "Late Chunking in Long-Context Embedding Models", Jina AI Blog, 2024-08-22. https://jina.ai/news/late-chunking-in-long-context-embedding-models/. Accessed 2026-05-25.
[^27]: MongoDB, "Contextualized Chunk Embeddings - Voyage AI by MongoDB", MongoDB Docs, 2025. https://www.mongodb.com/docs/voyageai/models/contextualized-chunk-embeddings/. Accessed 2026-05-25.
[^28]: NVIDIA, "Evaluating Retriever for Enterprise-Grade RAG", NVIDIA Technical Blog, 2024. https://developer.nvidia.com/blog/evaluating-retriever-for-enterprise-grade-rag/. Accessed 2026-05-25.
[^29]: AI21 Labs, "Chunk size is query-dependent: a simple multi-scale approach to RAG retrieval", AI21 Blog, 2024. https://www.ai21.com/blog/query-dependent-chunking/. Accessed 2026-05-25.