<p><strong>TLDR:</strong> Amazon Redshift’s new RG instances fold the Spectrum scanning fleet directly into the core engine, dropping the per-terabyte S3 fee and claiming up to 2.4x faster Iceberg queries over RA3. DuckDB Labs shipped Quack, a client-server protocol that finally lets multiple processes write to the same DuckDB database simultaneously. LangChain announced SmithDB at the Interrupt conference, a purpose-built database for agent trace data running on Apache DataFusion. PostgreSQL 19 keeps delivering: parallel autovacuum and a native REPACK command with online mode both landed.</p>
<hr>
<h2>Amazon Redshift RG instances</h2>
<p>The headline is the integrated data lake query engine. RA3 clusters used Redshift Spectrum as a separate scanning fleet with a $5/TB fee — RG folds that directly into the core engine, so Iceberg and Parquet queries in S3 run inside your VPC with no separate fleet and no per-terabyte charges. AWS claims up to 2.4x faster performance for Iceberg queries and 1.5x for Parquet versus RA3 with Spectrum, plus 30% lower price per vCPU. RG also bundles JIT Analyze for automatic statistics collection, intelligent NVMe caching, vectorized Parquet scans, and advanced partition-level pruning. Existing RA3 clusters can migrate via Elastic Resize (roughly 10-15 minutes of downtime), Snapshot and Restore, or Classic Resize. Available now in 26 regions as <code>rg.xlarge</code> and <code>rg.4xlarge</code>.</p>
<h2>DuckDB Quack protocol</h2>
<p>Quack is a DuckDB extension that routes writes through a single server process over HTTP, serializing access and removing the single-writer bottleneck that’s been the main complaint about DuckDB’s embedded architecture. Data moves using DuckDB’s internal vector block format, keeping serialization overhead low. Benchmarks show 60 million rows transferred in under 5 seconds, beating both PostgreSQL and Arrow Flight SQL for bulk transfers. For concurrent writes up to 8 threads it also outperforms PostgreSQL, though past 8 threads the picture gets more complicated. The auth model is basic shared tokens with no per-user permissions and no HTTPS by default — DuckDB recommends nginx in front for any internet exposure. A production release is planned alongside DuckDB v2.0 in fall 2026, with DuckLake integration and replication on the roadmap.</p>
<h2>LangChain SmithDB</h2>
<p>LangChain announced SmithDB at the Interrupt conference: a database purpose-built for agent observability workloads, built on Apache DataFusion, now powering core parts of LangSmith. The pitch is that agent traces have outgrown general-purpose databases — high write throughput, append-heavy workloads, and query patterns that don’t map well onto row stores. Worth watching if you’re building observability infrastructure for agentic systems, though the technical details beyond the DataFusion foundation are still sparse.</p>
<h2>PostgreSQL 19 parallel autovacuum and REPACK</h2>
<p>Two meaningful additions for anyone running write-heavy PostgreSQL workloads. Parallel autovacuum extends the parallel vacuum support from PG13 to background workers, letting them spawn parallel helpers for index vacuuming on large tables with multiple indexes. It’s disabled by default (<code>autovacuum_max_parallel_workers=0</code>) and most useful on tables with 5+ large indexes where index vacuuming dominates total vacuum time. REPACK consolidates VACUUM FULL and CLUSTER into a single command with a CONCURRENTLY mode that uses logical decoding to keep tables readable and writable throughout, with only a brief ACCESS EXCLUSIVE lock at the final swap — essentially pg_repack functionality in core. Tables need a primary key or replica identity; unlogged and partitioned tables aren’t supported yet.</p>
<hr>
<h2>Also notable</h2>
<ul>
<li><strong>Aurora Serverless performance update:</strong> AWS claims up to 45% faster scale-up during demand spikes and up to 30% performance gains through workload-aware scheduling.</li>
<li><strong>Amazon S3 Vectors (preview):</strong> Native vector embedding storage and similarity search built into S3, with up to 10,000 vector indexes per bucket and a claimed 90% cost reduction versus standalone vector databases. Currently in select regions including us-east-1.</li>
<li><strong>Netflix on Apache Druid:</strong> Interval-aware caching that decomposes rolling window queries into reusable time segments achieves an 84% cache hit rate and reduces query load by 33%.</li>
<li><strong>Databricks RG instances rate limiting:</strong> Databricks rebuilt their rate limiting service by moving counters in-memory with a sharded routing layer (Dicer), switching to async batch-reporting, and adopting token bucket — cutting tail latency roughly 10x at the cost of ~5% overshoot accuracy.</li>
<li><strong>Unity Catalog ABAC GA:</strong> Attribute-based access control for row filtering and column masking, governed tags, and automated data classification (including custom PII classifiers) are now generally available in Unity Catalog.</li>
<li><strong>Google Cloud proxy models:</strong> Lightweight classifiers trained on Gemini embeddings replace most LLM calls in BigQuery and AlloyDB AI functions, achieving 90-116% of LLM accuracy while consuming ~400x fewer tokens and reducing query latency 30-100x. Already live via ‘optimized mode’ for AI.IF and AI.CLASSIFY.</li>
<li><strong>MinIO MemKV:</strong> Flash-based context memory store accessed via 800GbE RDMA, claiming 95%+ better GPU utilization by eliminating KV cache recompute. Enables stateless serving layers where any GPU replica can resume a conversation mid-flight.</li>
<li><strong>pg_column_tetris:</strong> A pure SQL/PL/pgSQL PostgreSQL extension that enforces optimal column ordering to eliminate row padding waste, with warn, strict, and off modes. Works on PostgreSQL 14+ including RDS, Cloud SQL, Supabase, and Neon.</li>
<li><strong>pgrwl:</strong> A Go single-binary tool for cloud-native continuous PostgreSQL backup using the streaming replication protocol directly (not archive_command), with WAL streaming, AES-256-GCM encryption, and RPO=0 support via replication slots.</li>
<li><strong>Amazon RDS for Oracle M8i/R8i:</strong> New instance types with custom Intel Xeon 6 processors, claiming up to 15% better price-performance and 2.5x more memory bandwidth versus previous-generation Intel instances, now available with SE2 License Included pricing.</li>
<li><strong>Debezium community meetings:</strong> Weekly team meetings open to the public starting May 19, every Tuesday for up to 45 minutes. Agendas maintained in a shared Google Doc with topics proposable via Zulip.</li>
<li><strong>Elastic Stack 9.4.1:</strong> Bug fix release superseding 9.4.0, no major feature additions.</li>
</ul>


Query optimization, schema design, and data store coverage across PostgreSQL, MongoDB, Redis, and more.

Databases Digest

A roundup of notable database and data infrastructure updates. Amazon Redshift's new RG instances integrate the Spectrum scanning fleet into the core engine, eliminating per-terabyte S3 fees and claiming up to 2.4x faster Iceberg query performance over RA3. DuckDB Labs shipped Quack, a client-server extension enabling concurrent multi-process writes via HTTP, with a production release planned for DuckDB v2.0 in fall 2026. LangChain announced SmithDB at the Interrupt conference, a purpose-built agent observability database built on Apache DataFusion. PostgreSQL 19 gains parallel autovacuum for background workers and a native REPACK command with online mode. Additional highlights include Aurora Serverless performance improvements, Amazon S3 Vectors preview for native vector storage, Netflix's interval-aware Druid caching achieving 84% cache hit rates, and several PostgreSQL tooling releases including pg_column_tetris and pgrwl.

Amazon Redshift gets a new instance family, DuckDB ships concurrent write support