We are thrilled to announce the open-source release of <a href="https://github.com/cocoindex-io/cocoindex" target="_blank" rel="noopener nofollow">CocoIndex</a>, an ETL to to prepare data for AI, such as semantic search, retrieval-augmented generation (RAG). CocoIndex is open source under the Apache License 2.0.
Github: <a href="https://github.com/cocoindex-io/cocoindex" target="_blank" rel="noopener nofollow">https://github.com/cocoindex-io/cocoindex</a>
It is world’s first engine that supports both custom transformation logic (like building lego) and incremental updates (out of box, to handle source data updates) for indexing data.
<img src="https://media.daily.dev/image/upload/s--WuRlvXHq--/f_auto/v1741845013/ugc/content_d0265ca1-405d-4d9c-aba8-507eee443aef" alt="Screenshot 2025-03-10 at 10.20.45 PM">
CocoIndex offers a data-driven programming model that simplifies the creation and maintenance of data indexing pipelines, ensuring data freshness and consistency.
<h2>🔥 Key Features</h2>
<ul>
<li>
Data Flow Programming: Build indexing pipelines by composing transformations like Lego blocks, with built-in state management and observability.
</li>
<li>
Support Custom Logic: Plug in your choice of chunking, embedding, and vector stores. Extend with custom transformations like deduplication and reconciliation.
</li>
<li>
Incremental Updates: Smart state management minimizes re-computation by tracking changes at the file level, with future support for chunk-level granularity.
</li>
<li>
Python SDK: Built with a RUST core 🦀 for performance, exposed through an intuitive Python binding 🐍 for ease of use.
We are moving fast and a lot of features and improvements are coming soon.
</li>
</ul>
How it works:
You can think of an indexing flow similar to formulas in a spreadsheet:
<ul>
<li>In a spreadsheet, you define formulas that transform input cells into output cells</li>
<li>When input values change, the spreadsheet automatically recalculates affected outputs</li>
<li>You focus on defining the transformation logic, not managing updates</li>
</ul>
CocoIndex works the same way, but with more powerful capabilities:
<ul>
<li>Instead of flat tables, CocoIndex models data in nested data structures, making it more natural to model complex data</li>
<li>Instead of simple cell-level formulas, you have operations like “for each” to apply the same formula across rows without repeating yourself</li>
</ul>
For example:
<img src="https://media.daily.dev/image/upload/s--DKtdWihS--/f_auto/v1741845506/ugc/content_cbf9150a-04b3-4517-9a9d-04165fec0cfa" alt="Screenshot 2025-03-05 at 7.30.32 PM">
Takes 90 lines of python to declare the flow
<pre><code>@cocoindex.flow_def(name=&quot;TextEmbedding&quot;)
def text_embedding(flow: cocoindex.FlowBuilder, data: cocoindex.DataScope):
 # Add data source.
 data[&quot;documents&quot;] = flow.add_source(
 cocoindex.sources.LocalFile(path=&quot;sourceFiles&quot;))
 doc_embeddings = data.add_collector()

 with data[&quot;documents&quot;].row() as doc:
 # Split into chunks.
 doc[&quot;chunks&quot;] = doc[&quot;content&quot;].transform(
 cocoindex.functions.SplitRecursively(chunk_size=512))

 with doc[&quot;chunks&quot;].row() as chunk:
 # Embed each chunk.
 chunk[&quot;embedding&quot;] = chunk[&quot;text&quot;].transform(
 cocoindex.functions.SentenceTransformerEmbed(
 model=&quot;sentence-transformers/all-MiniLM-L6-v2&quot;)

 # Collect embedding and metadata for indexing.
 doc_embeddings.collect(
 filename=doc[&quot;filename&quot;], location=chunk[&quot;location&quot;],
 text=chunk[&quot;text&quot;], embedding=chunk[&quot;embedding&quot;])

 # Export to vector store.
 doc_embeddings.export(
 &quot;doc_embeddings&quot;, cocoindex.storages.Postgres(),
 primary_key_fields=[&quot;filename&quot;, &quot;location&quot;],
 vector_index=[&quot;embedding&quot;])
</code></pre>
For a detailed walkthrough, refer to our <a href="https://cocoindex.io/docs/getting_started/quickstart/" target="_blank" rel="noopener nofollow">Quick Start Guide</a>
Here is a <a href="https://youtu.be/gv5R8nOXsWU?si=tyRTQCthv9WgOxoT" target="_blank" rel="noopener nofollow"> Video Guide</a>.
<h2>🤗 Community</h2>
We are super excited for community contributions of all kinds - whether it’s code improvements, documentation updates, issue reports, feature requests on GitHub, and discussions in our Discord.
Looking forward to learn from you and see you there! :)

The official Open Source daily.dev Squad. 

Open Source

CocoIndex is an open-source ETL tool designed to prepare data for AI applications such as semantic search and retrieval-augmented generation. It features a data-driven programming model, custom transformation logic, and incremental updates. Built on a Rust core with a Python SDK, CocoIndex allows users to build indexing pipelines using a modular, Lego-like approach, ensuring data consistency and minimal re-computation.

CocoIndex - ETL to prepare fresh data for AI, like LEGO