Repo: <a href="https://github.com/cocoindex-io/cocoindex" target="_blank" rel="noopener nofollow">https://github.com/cocoindex-io/cocoindex</a>
(Appreciate a star if it is helpful!)
In this project, we will build image search and query it with natural language. You can search for “a cute animal” or “a red car”, and the system returns visually relevant results — no manual tagging needed.
We are going to use multi-modal embedding model CLIP to understand and directly embed the image; and build a vector index for efficient retrieval. We are going use CocoIndex to build the indexing flow. It supports long running flow and only process changed files - we can keep adding new files to the folder and it will be indexed within a minute.
<ol>
<li>Ingest the images.</li>
</ol>
<pre><code>@cocoindex.flow_def(name=&quot;ImageObjectEmbedding&quot;)
def image_object_embedding_flow(flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope):
 data_scope[&quot;images&quot;] = flow_builder.add_source(
 cocoindex.sources.LocalFile(path=&quot;img&quot;, included_patterns=[&quot;*.jpg&quot;, &quot;*.jpeg&quot;, &quot;*.png&quot;], binary=True),
 refresh_interval=datetime.timedelta(minutes=1) # Poll for changes every 1 minute
 )
 img_embeddings = data_scope.add_collector()
</code></pre>
flow_builder.add_source will create a table with sub fields (filename, content), we can refer to the documentation for more details.
<img src="https://media.daily.dev/image/upload/s--1rCtmn9v--/f_auto/v1750054498/ugc/content_bf01d3c5-8f4a-4273-a311-324f60e585ac?_a=BAMClqUq0" alt="Screenshot 2025-06-15 at 11.14.55 PM">
<ol start="2">
<li>Process each image and collect the information.
2.1 Embed the image with CLIP</li>
</ol>
<pre><code>@functools.cache
def get_clip_model() -&gt; tuple[CLIPModel, CLIPProcessor]:
 model = CLIPModel.from_pretrained(CLIP_MODEL_NAME)
 processor = CLIPProcessor.from_pretrained(CLIP_MODEL_NAME)
 return model, processor
</code></pre>
The @functools.cache decorator caches the results of a function call. In this case, it ensures that we only load the CLIP model and processor once.
<pre><code>@cocoindex.op.function(cache=True, behavior_version=1, gpu=True)
def embed_image(img_bytes: bytes) -&gt; cocoindex.Vector[cocoindex.Float32, Literal[384]]:
 &quot;&quot;&quot;
 Convert image to embedding using CLIP model.
 &quot;&quot;&quot;
 model, processor = get_clip_model()
 image = Image.open(io.BytesIO(img_bytes)).convert(&quot;RGB&quot;)
 inputs = processor(images=image, return_tensors=&quot;pt&quot;)
 with torch.no_grad():
 features = model.get_image_features(**inputs)
 return features[0].tolist()
</code></pre>
embed_image is a custom function that uses the CLIP model to convert an image into a vector embedding. It accepts image data in bytes format and returns a list of floating-point numbers representing the image’s embedding.
The function supports caching through the cache parameter. When enabled, the executor will store the function’s results for reuse during reprocessing, which is particularly useful for computationally intensive operations. For more information about custom function parameters, refer to the documentation.
Then we are going to process each image and collect the information.
<pre><code>with data_scope[&quot;images&quot;].row() as img:
 img[&quot;embedding&quot;] = img[&quot;content&quot;].transform(embed_image)
 img_embeddings.collect(
 id=cocoindex.GeneratedField.UUID,
 filename=img[&quot;filename&quot;],
 embedding=img[&quot;embedding&quot;],
 )
</code></pre>
<img src="https://media.daily.dev/image/upload/s--r6_xuaCG--/f_auto/v1750054561/ugc/content_109dd6ea-89e8-4d8d-8f59-5521b17dd537?_a=BAMClqUq0" alt="Screenshot 2025-06-15 at 11.15.57 PM">
2.3 Collect the embeddings
Export the embeddings to a table in Qdrant.
<pre><code>img_embeddings.export(
 &quot;img_embeddings&quot;,
 cocoindex.storages.Qdrant(
 collection_name=&quot;image_search&quot;,
 grpc_url=QDRANT_GRPC_URL,
 ),
 primary_key_fields=[&quot;id&quot;],
 setup_by_user=True,
)
</code></pre>
<ol start="3">
<li>Query the index
Embed the query with CLIP, which maps both text and images into the same embedding space, allowing for cross-modal similarity search.</li>
</ol>
<pre><code>def embed_query(text: str) -&gt; list[float]:
 model, processor = get_clip_model()
 inputs = processor(text=[text], return_tensors=&quot;pt&quot;, padding=True)
 with torch.no_grad():
 features = model.get_text_features(**inputs)
 return features[0].tolist()
</code></pre>
Defines a FastAPI endpoint /search that performs semantic image search.
<pre><code>@app.get(&quot;/search&quot;)
def search(q: str = Query(..., description=&quot;Search query&quot;), limit: int = Query(5, description=&quot;Number of results&quot;)):
 # Get the embedding for the query
 query_embedding = embed_query(q)
 
 # Search in Qdrant
 search_results = app.state.qdrant_client.search(
 collection_name=&quot;image_search&quot;,
 query_vector=(&quot;embedding&quot;, query_embedding),
 limit=limit
 )
</code></pre>
This searches the Qdrant vector database for similar embeddings. Returns the top limit results
<pre><code># Format results
out = []
for result in search_results:
 out.append({
 &quot;filename&quot;: result.payload[&quot;filename&quot;],
 &quot;score&quot;: result.score
 })
return {&quot;results&quot;: out}
</code></pre>
#Time to have fun!
Setup indexing flow
<pre><code>cocoindex setup main.py
</code></pre>
It is setup with a live updater, so you can add new files to the folder and it will be indexed within a minute.
Run backend
<pre><code>uvicorn main:app --reload --host 0.0.0.0 --port 8000
</code></pre>
Run frontend
<pre><code>cd frontend
npm install
npm run dev
</code></pre>
<img src="https://media.daily.dev/image/upload/s--urviKRBf--/f_auto/v1750054700/ugc/content_f8e05bb2-5b7d-4da1-bc79-6ce7c3322509?_a=BAMClqUq0" alt="Screenshot 2025-06-15 at 11.18.15 PM">

Star

The official daily.dev AI community. Led by thought leaders and AI experts. Stay tuned for insights, news and discussions. 

A step-by-step guide to building an image search system using CLIP embeddings and vector indexing. The tutorial demonstrates how to create a multi-modal search engine that accepts natural language queries like 'a cute animal' and returns visually relevant images without manual tagging. The implementation uses CocoIndex for data processing, CLIP for generating embeddings, Qdrant for vector storage, and FastAPI for the search API, all contained within 200 lines of Python code.

Build Image Search and run on your PC within 200 lines of python

Nice. Building tools on your own is the future