<p><strong>TLDR:</strong> ClickHouse Cloud’s new index sharding feature cuts memory pressure dramatically at petabyte scale, with benchmark gains up to 7.7x on bloom filter lookups. Airbnb published a detailed writeup on their homegrown metrics storage system handling 50 million samples per second across 1.3 billion active time series. PostgreSQL continues to dominate the conversation today, from RLS multi-tenancy debates to a CONCURRENTLY option landing in PG19 for REPACK. Snowflake had a busy day too, announcing Cortex Agents, Cortex Code expansions, and Claude Opus 4.7 availability on their platform.</p>
<hr>
<h2>ClickHouse index sharding in Cloud</h2>
<p>ClickHouse Cloud is rolling out index sharding for SharedMergeTree tables in private preview. The problem it solves is straightforward: at petabyte scale, every replica was loading the full index into working memory — up to 400 GiB per replica. With sharding, each replica owns a 1/N slice via consistent hashing, so total fleet-wide index memory stays flat regardless of how many replicas you add. Benchmarks on a 50 billion row table show 7.7x faster bloom filter lookups, 7.2x for vector search, and 4.3x for primary key range queries. It activates automatically once tables exceed 10 parts and 1 GB of index data.</p>
<h2>Airbnb’s metrics storage at 50M samples per second</h2>
<p>Airbnb’s engineering team published a detailed account of building and operating their own metrics storage system — 50 million samples per second, 2.5 petabytes of logical time series data, 1.3 billion active time series. The interesting engineering is in the multi-tenancy story: shuffle sharding for blast radius isolation, per-tenant guardrails on reads and writes, and compaction sharding for large tenants that would otherwise starve smaller ones. They use Promxy for federated cross-cluster queries, but flag that federated queries run 5-10x more expensive than single-cluster queries — worth keeping in mind if you’re designing anything similar.</p>
<h2>PostgreSQL REPACK gets CONCURRENTLY in PG19</h2>
<p>PostgreSQL 19 is adding a CONCURRENTLY option to REPACK, which means you can reclaim table bloat without holding an access-exclusive lock for the entire operation. A test on a 200M-row table (19 GB, 80% bloat) shrank it to ~4 GB in about 65 seconds, with concurrent inserts seeing average latency climb from 1.5ms to 5.5ms rather than blocking entirely. Known limitations: it consumes a replication slot, only one concurrent REPACK can run system-wide, and there’s a deadlock risk during the final lock upgrade. All three are flagged for follow-up commits.</p>
<h2>PostgreSQL as a drop-in for Redis, RabbitMQ, and Elasticsearch</h2>
<p>A benchmark post compared naive versus optimized PostgreSQL 17 patterns across caching, job queues, full-text search, and pub/sub. The headline numbers: tsvector with GIN indexes delivers a 10x p95 improvement over ILIKE; SKIP LOCKED is production-ready for job queues; UNLOGGED tables give a consistent 13% write speedup; LISTEN/NOTIFY cuts pub/sub latency by 20% at p95. Under combined workload the optimized setup adds ~15% overhead but stays under 10ms p95. The conclusion — that a well-tuned PostgreSQL instance can replace 3-5 specialized services for most web apps — is reasonable, though the post is careful to note where dedicated tools still make sense.</p>
<hr>
<h2>Also notable</h2>
<ul>
<li><strong>Snowflake Cortex Agents GA:</strong> Snowflake announced Cortex Agents with MCP connectors for Jira, Salesforce, GitHub, and Slack, plus agent versioning with instant rollback and resource budgets for cost control at the agent level.</li>
<li><strong>Claude Opus 4.7 on Snowflake Cortex:</strong> Anthropic’s latest model is now in public preview on Cortex AI in the US and EU, with adjustable effort levels for balancing performance against latency.</li>
<li><strong>Cortex Code expands:</strong> Snowflake’s AI coding agent now supports AWS Glue, Databricks, and Postgres alongside existing dbt and Airflow support, with a VS Code extension, MCP server, and Claude Code plugin.</li>
<li><strong>Redpanda Shadow Linking:</strong> Redpanda’s broker-native DR feature runs inside the broker in C++, preserves offsets exactly, and benchmarks at 2.5 GiB/s with sub-10,000 message lag (~4ms RPO) — no JVM tuning, no extra hardware.</li>
<li><strong>MongoDB pipeline ordering matters:</strong> A production incident traced a 600ms-to-17s regression to a $lookup placed before aggregation and $limit, causing 2.5M nested loop joins. Reordering to limit first, then lookup, brought it back to 0.8s.</li>
<li><strong>ClickHouse ClickPipes GA:</strong> Terraform and OpenAPI support for ClickPipes is now generally available from provider v3.14.0, including CDC connectors for Postgres, MySQL, and MongoDB, plus BigQuery and Azure Blob Storage.</li>
<li><strong>Tigris chaos testing:</strong> Using Antithesis for deterministic fault injection, Tigris found three cache coherence bugs over nine months — all sharing the same root cause: a window between a FoundationDB metadata commit and edge cache invalidation.</li>
<li><strong>GPU monitoring cardinality:</strong> A 1,000-node GPU cluster with 8 GPUs each generates 1.4M unique time series from hardware metrics alone. The practical fix is routing high-cardinality fields (pod names, job IDs, model versions) to logs rather than metrics labels.</li>
<li><strong>PostgreSQL RLS for multi-tenancy:</strong> A developer compared Eloquent scoping, database-per-tenant, and RLS in Laravel, landing on RLS as the right call for solo devs and small teams — though it breaks standard Laravel testing traits and rules out SQLite for tests.</li>
<li><strong>Aurora Serverless v4:</strong> Amazon’s new platform version claims up to 30% better performance with an improved scaling algorithm for bursty workloads; all new clusters launch on v4 automatically.</li>
<li><strong>Polish full-text search in PostgreSQL:</strong> A practical guide covers setting up Hunspell-based Polish FTS config today, with a note that PostgreSQL 19 will include a built-in Polish configuration out of the box.</li>
<li><strong>Databricks Excel Add-in:</strong> Public preview of an Excel Add-in built on Databricks SQL and Unity Catalog, reducing lakehouse data access setup from dozens of manual steps to a few clicks for business users.</li>
</ul>


Query optimization, schema design, and data store coverage across PostgreSQL, MongoDB, Redis, and more.

Databases Digest

A roundup of major database and infrastructure news: ClickHouse Cloud introduces index sharding for SharedMergeTree tables, cutting per-replica memory from 400 GiB and delivering up to 7.7x faster bloom filter lookups at petabyte scale. Airbnb details their homegrown metrics storage system handling 50M samples/sec across 1.3B active time series, with shuffle sharding for multi-tenancy isolation. PostgreSQL 19 gains a CONCURRENTLY option for REPACK, enabling bloat reclamation without exclusive locks. A benchmark post shows optimized PostgreSQL 17 can replace Redis, RabbitMQ, and Elasticsearch for most web apps. Snowflake announces Cortex Agents GA with MCP connectors, Claude Opus 4.7 availability, and Cortex Code expansions. Additional items cover Redpanda Shadow Linking, a MongoDB pipeline ordering production incident, Aurora Serverless v4, and GPU monitoring cardinality challenges.

ClickHouse ships index sharding, Airbnb's metrics store hits 50M samples per second