Project is open sourced - <a href="https://github.com/cocoindex-io/realtime-codebase-indexing" target="_blank" rel="noopener nofollow">https://github.com/cocoindex-io/realtime-codebase-indexing
</a>
Modern teams rely on AI coding agents, semantic search, and automated reviews, but most workflows still treat the codebase as static text blobs instead of a rich, structured knowledge graph. This creates brittle context windows, stale embeddings, and expensive full re-indexing jobs whenever the code changes.
CocoIndex addresses this by turning your repository into a continuously updated vector index, built directly from the filesystem, parsed with Tree-sitter, and exposed via SQL and Postgres’ pgvector. The result is a real-time semantic layer over your codebase that can drive agents, editors, and internal tools without constantly re-plumbing pipelines.
The flow is logically split into five stages:
<img src="https://media.daily.dev/image/upload/s--S4LAt_q6--/f_auto/v1768634235/ugc/content_6abdee69-b51e-4fc4-a2e1-442f9e186514?_a=BAMAMiiu0" alt="image">
​
<ul>
<li>Read code files from the local filesystem using a LocalFile source.</li>
<li>Derive language from file extensions so Tree-sitter can parse correctly.</li>
<li>Recursively split code into semantic chunks with SplitRecursively.</li>
<li>Generate dense embeddings per chunk with a SentenceTransformer model.</li>
<li>Export everything into Postgres with a vector index on the embedding column</li>
</ul>
<pre><code>@cocoindex.flow_def(name=&quot;CodeEmbedding&quot;)
def code_embedding_flow(
 flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope
) -&gt; None:
 &quot;&quot;&quot;
 Define an example flow that embeds files into a vector database.
 &quot;&quot;&quot;
 data_scope[&quot;files&quot;] = flow_builder.add_source(
 cocoindex.sources.LocalFile(
 path=os.path.join(&quot;..&quot;, &quot;..&quot;),
 included_patterns=[&quot;*.py&quot;, &quot;*.rs&quot;, &quot;*.toml&quot;, &quot;*.md&quot;, &quot;*.mdx&quot;],
 excluded_patterns=[&quot;**/.*&quot;, &quot;target&quot;, &quot;**/node_modules&quot;],
 )
 )
 code_embeddings = data_scope.add_collector()

 with data_scope[&quot;files&quot;].row() as file:
 file[&quot;language&quot;] = file[&quot;filename&quot;].transform(
 cocoindex.functions.DetectProgrammingLanguage()
 )
 file[&quot;chunks&quot;] = file[&quot;content&quot;].transform(
 cocoindex.functions.SplitRecursively(),
 language=file[&quot;language&quot;],
 chunk_size=1000,
 min_chunk_size=300,
 chunk_overlap=300,
 )
 with file[&quot;chunks&quot;].row() as chunk:
 chunk[&quot;embedding&quot;] = chunk[&quot;text&quot;].call(code_to_embedding)
 code_embeddings.collect(
 filename=file[&quot;filename&quot;],
 location=chunk[&quot;location&quot;],
 code=chunk[&quot;text&quot;],
 embedding=chunk[&quot;embedding&quot;],
 start=chunk[&quot;start&quot;],
 end=chunk[&quot;end&quot;],
 )

 code_embeddings.export(
 &quot;code_embeddings&quot;,
 cocoindex.targets.Postgres(),
 primary_key_fields=[&quot;filename&quot;, &quot;location&quot;],
 vector_indexes=[
 cocoindex.VectorIndexDef(
 field_name=&quot;embedding&quot;,
 metric=cocoindex.VectorSimilarityMetric.COSINE_SIMILARITY,
 )
 ],
 )
 
</code></pre>
<h2>Real-world use cases unlocked</h2>
Once a codebase is indexed this way, it becomes a substrate for an entire ecosystem of tooling.
<ul>
<li>
Semantic code context for AI coding agents (Claude, Codex, Gemini CLI, etc.), where each tool can pull high-precision context windows tailored to current edits or tasks.
</li>
<li>
MCP-style integrations for editors like Cursor, Windsurf, and VSCode, enabling offline or self-hosted semantic search and navigation.
</li>
<li>
Context-aware code search and natural language retrieval, letting developers ask “Where do we validate OAuth tokens?” instead of guessing filenames.
</li>
<li>
AI-powered code review and PR summarization, drawing from related chunks and design areas, not just the diff.
</li>
<li>
Automated refactoring and large-scale migrations that need to understand usage patterns across the entire repository.
</li>
<li>
SRE workflows that index infrastructure-as-code, deployment scripts, and config files for fast incident root cause and blast-radius analysis.
</li>
</ul>
Continuous generation of design documentation that stays in sync with implementation by querying the index and synthesizing higher-level artifacts.
​

CocoIndex is an open-source tool that transforms codebases into continuously updated vector indexes using filesystem monitoring, Tree-sitter parsing, and Postgres pgvector. The system reads source files, detects languages, splits code into semantic chunks, generates embeddings, and stores them in a searchable database. This enables real-time semantic search, AI agent context retrieval, code review automation, and infrastructure analysis without expensive re-indexing. The five-stage pipeline handles file reading, language detection, recursive splitting, embedding generation, and Postgres export with vector indexing.

Real-Time Codebase Indexing: From Source Files to Searchable Context