Jina AI Introduced ‘Late Chunking’: A Simple AI Approach to Embed Short Chunks by Leveraging the Power of Long-Context Embedding Models

Retrieval-augmented generation (RAG) involves breaking large documents into smaller text chunks for efficient information retrieval using embedding models. The release of jina-embeddings-v2-base-en, an open-source model with 8K context length, highlighted practical limitations in handling long documents. Late Chunking, a new approach, addresses these issues by applying the transformer layer to the whole text first, preserving contextual information and improving retrieval efficiency. Tests showed that late chunking outperforms traditional methods, especially for longer texts.