How Airtable Built the Search Layer Behind Their AI Features

Airtable's engineering team built a semantic search layer to power AI features like natural-language querying (Omni) and linked record recommendations. The core challenge was managing embeddings for hundreds of thousands of customer bases, 75% of which are idle at any given time. Key decisions included: using Milvus as the self-hosted vector database with one partition per customer base for strong isolation; implementing hierarchical capping (400 collections × 1,000 partitions per cluster) to avoid performance degradation beyond 100K partitions; choosing HNSW indexing for its low latency and high recall despite higher memory cost; and using hot/cold partition offloading to reduce memory usage given the bursty, bimodal access pattern. Recovery is handled by re-embedding from source using the existing async pipeline rather than traditional backup/restore. Every architectural decision traces back to the specific properties of Airtable's data.