<p><strong>TLDR:</strong> Manticore Search 25.0.0 is out with hybrid search, KNN prefiltering, parallel chunk merging, and S3-native backups in a single release. Databricks published data showing AI agents now create 4x more databases than humans in Lakebase, which says something real about where agentic workloads are headed. ClickHouse 26.3 LTS lands with materialized CTEs, async inserts on by default, and a new bucketed Map storage format. Netflix shared how they cut Druid query volume by 33% with an interval-aware caching layer built on Cassandra.</p>
<hr>
<h2>Manticore Search 25.0.0</h2>
<p>This is a substantial release. Hybrid search (BM25 + KNN via Reciprocal Rank Fusion) and KNN prefiltering land together, which matters because postfiltering on selective attribute filters has always been the quiet failure mode of vector search — you ask for k results and get back fewer. The prefilter implementation uses three strategies automatically: standard filtered HNSW, ACORN-1 for low-selectivity cases, and brute-force linear scan when the filtered subset is small enough that graph traversal isn’t worth it. Parallel chunk merging cuts RT table compaction time by 55% on 10M-document benchmarks, with columnar storage seeing up to 6.8x improvement. S3-compatible backup now streams directly without intermediate local files, supporting AWS, MinIO, and Cloudflare R2.</p>
<h2>Agentic workloads are changing what databases need to be</h2>
<p>Databricks published a post on how agentic development is reshaping database requirements, and the numbers are worth sitting with. AI agents in Lakebase now create roughly 4x more databases than human users. Half of agentic app databases have compute lifetimes under 10 seconds, which makes fixed-cost database pricing completely unworkable. The post covers three requirements that fall out of this: O(1) copy-on-write branching for rapid iteration, sub-second scale-to-zero elasticity, and open storage formats so agents trained on Postgres ecosystems don’t hallucinate. Neon published related work on zero-downtime patching via prewarming — spinning up a new compute node, populating its cache from shared storage, and subscribing to WAL before promoting it, with benchmarks showing near-instant throughput recovery versus a 70% TPS drop on cold restart.</p>
<h2>ClickHouse 26.3 LTS</h2>
<p>Materialized CTEs are the headline: subqueries evaluate once and land in temporary tables, with benchmarks showing roughly 2x speedup for queries that reference the same CTE multiple times. Async inserts are now on by default, which removes a configuration step that tripped up a lot of small-write workloads. The new bucketed Map storage format is worth attention for anyone using Map columns in observability schemas — keys are split into hash-based buckets during background merges, so single-key lookups scan dramatically less data. TTL DELETE operations can now use vertical merges, which reduces memory pressure on wide-table workloads.</p>
<h2>Netflix’s interval-aware Druid cache</h2>
<p>The core insight here is simple and generalizable: in a rolling time window, most of the data is already settled. Netflix built a two-level cache keyed by query hash (without the time interval) and bucketed timestamps, with exponential TTLs that grow with data age. On a partial cache hit, only the missing tail goes to Druid. The results are concrete — 82% partial hit rate, 84% of result data served from cache, 33% reduction in Druid query volume, and 66% improvement in P90 latency. The cache is backed by Cassandra via Netflix’s KVDAL. If you’re running any time-series database with overlapping rolling-window queries, this pattern applies directly.</p>
<hr>
<h2>Also notable</h2>
<ul>
<li><strong>Databricks Unity Catalog Business Semantics GA:</strong> Centrally defined KPI metric views are now generally available, with the core Metric Views implementation being open sourced in Apache Spark OSS and Unity Catalog OSS v0.5 support coming soon.</li>
<li><strong>Databricks Lakeflow Connect free tier:</strong> Every workspace now gets 100 free DBUs/day for managed SaaS and database connectors, covering up to 100 million records/day across 30+ connectors including Postgres, MySQL, Snowflake, and BigQuery.</li>
<li><strong>Databricks Default Warehouse GA:</strong> Workspace admins can now set a default SQL warehouse for ad hoc queries via settings or API, with per-user override support — over 300 customers already using it to prevent accidental production warehouse wakeups.</li>
<li><strong>Figma FigCache:</strong> Figma replaced their fragmented Redis client ecosystem with a stateless RESP-wire-protocol proxy using a Starlark-based configuration system for dynamic routing; the result is six nines of uptime and zero-downtime failovers.</li>
<li><strong>Nextdoor scaling ladder:</strong> A detailed writeup on their PostgreSQL evolution — PgBouncer, primary-replica with dynamic routing, Valkey cache with MessagePack and Zstd, PostgreSQL trigger-based cache versioning with atomic Lua compare-and-set, and Debezium CDC for reconciliation.</li>
<li><strong>SQLite in production data loss:</strong> A Rails 8 e-commerce team lost two paid orders when rapid-fire Kamal deploys caused overlapping containers to corrupt SQLite writes — WAL mode doesn’t protect you from concurrent processes on the same file.</li>
<li><strong>Respan ClickHouse migration:</strong> Outgrew Postgres at 50-100 writes/second, migrated to ClickHouse Cloud with incremental materialized views at minute and hourly rollups; now handling 50 million daily events without constant schema rework.</li>
<li><strong>Rill + ClickHouse:</strong> Rill processes 100B+ daily events for Bloomberg, AT&amp;T, and Comcast using ClickHouse as the analytical backend with a metrics-as-code semantic layer defined in SQL and YAML.</li>
<li><strong>PostgresBench:</strong> ClickHouse released an open, reproducible pgbench-based benchmark comparing five managed Postgres services at 100GB and 500GB scale with 256 concurrent clients; all configs and results are public.</li>
<li><strong>Cerbos Synapse for Trino:</strong> Row-level security, column masking, and table-level access control for Apache Trino via a protocol translation layer between Trino’s OPA plugin and Cerbos PDP, with policies managed centrally in Cerbos Hub.</li>
<li><strong>MakeMyTrip RTM migration:</strong> Switched from Spark micro-batch to Databricks Real-Time Mode with a single line of code change, dropping hotel personalization latency from 1-2 seconds to sub-50ms and seeing a 7% CTR boost.</li>
<li><strong>Neon + Stripe Projects:</strong> Running <code>stripe projects add neon</code> now provisions a Postgres database and returns credentials automatically, targeting the gap where agents could write code but couldn’t set up real infrastructure.</li>
<li><strong>TigerFS:</strong> A FUSE/NFS filesystem backed by PostgreSQL that lets you interact with databases using standard Unix tools; supports pipeline queries via path chaining and atomic multi-agent task coordination using directory moves.</li>
<li><strong>InfluxDB 3.9:</strong> Expanded CLI support for headless environments, 1MB default string field limit, DataFusion upgrades, and a beta performance preview with optimized single-series queries and automatic distinct value caches.</li>
<li><strong>Discord-style hybrid architecture:</strong> A post on chat application database design validates the PostgreSQL-for-metadata plus Cassandra/ScyllaDB-for-messages pattern, citing Discord’s migration at trillions of messages as the reference case.</li>
</ul>


Query optimization, schema design, and data store coverage across PostgreSQL, MongoDB, Redis, and more.

Databases Digest

Manticore Search 25.0.0 ships with hybrid BM25+KNN search, KNN prefiltering using three automatic strategies (filtered HNSW, ACORN-1, brute-force), 55% faster RT table compaction via parallel chunk merging, and S3-native streaming backups. Databricks data shows AI agents create 4x more databases than humans in Lakebase, with half of agentic databases having compute lifetimes under 10 seconds — driving requirements for copy-on-write branching, sub-second scale-to-zero, and open storage formats. ClickHouse 26.3 LTS introduces materialized CTEs (2x speedup for repeated references), async inserts on by default, and a bucketed Map storage format for faster single-key lookups. Netflix cut Druid query volume 33% and improved P90 latency 66% with an interval-aware two-level cache backed by Cassandra, achieving 82% partial hit rates on rolling time-window queries. Additional coverage includes Figma's Redis proxy replacement, Nextdoor's PostgreSQL scaling journey, SQLite data loss from concurrent Kamal deploys, and Neon's Stripe integration for agent-driven database provisioning.

Manticore Search 25.0 ships, agentic workloads are reshaping database design