<p><strong>TLDR:</strong> Apache Iceberg V3’s Deletion Vectors cut write overhead by up to 4,358x compared to rewriting Parquet files, and Snowflake now supports bidirectional interoperability with external catalogs like AWS Glue and Unity Catalog. SQL Server 2025 ships native regex functions that finally replace the CHARINDEX/PATINDEX mess. Debezium 3.4.3 patches a silent data corruption bug in the PostgreSQL pgoutput plugin when multibyte UTF-8 names are involved. Padlet is running 8 billion events per month through ClickHouse Cloud with a median query latency of 45ms.</p>
<hr>
<h2>Apache Iceberg V3 and Snowflake interoperability</h2>
<p>Iceberg V3 introduces Row Lineage and Deletion Vectors, and the Deletion Vectors story is the one worth paying attention to: instead of rewriting entire Parquet files on delete, the engine writes a small deletion bitmap, reducing write amplification by up to 4,358x in the benchmarks cited. Snowflake now exposes two interoperability paths — Catalog-Linked Databases for connecting external catalogs, and Horizon IRC for exposing Snowflake-managed tables to Spark, Trino, and Flink — both sharing the same RBAC and masking policy layer. CDC streams on externally managed Iceberg tables are now possible for the first time, which opens up Dynamic Iceberg Tables as a practical option rather than a theoretical one.</p>
<h2>SQL Server 2025 native regex</h2>
<p>SQL Server 2025 and Azure SQL Database ship six native T-SQL regex functions: <code>regexp_like</code>, <code>regexp_count</code>, <code>regexp_replace</code>, <code>regexp_substr</code>, <code>regexp_split_to_table</code>, and <code>regexp_matches</code>. The syntax is compatible with Java and C# regex, so patterns are portable. The most immediately useful addition is <code>regexp_split_to_table</code> with multi-character delimiters, which has been a genuine gap in <code>STRING_SPLIT</code> for years. <code>regexp_matches</code> returns position metadata and JSON output, which is useful for extraction pipelines that currently rely on ugly nested <code>SUBSTRING</code>/<code>CHARINDEX</code> chains.</p>
<h2>Debezium 3.4.3 patches silent PostgreSQL corruption</h2>
<p>This one deserves attention before you skip past it: Debezium 3.4.3 fixes a silent data corruption bug in the PostgreSQL pgoutput plugin triggered by multibyte UTF-8 characters in table or column names. Silent corruption is the worst kind — no error, no alert, just wrong data downstream. The release also includes a LogMiner I/O improvement for the Oracle connector that computes precise SCN boundaries before loading redo logs, which should reduce startup overhead on large Oracle deployments. Eight issues total were resolved.</p>
<h2>Padlet’s ClickHouse Cloud pipeline</h2>
<p>Padlet (40 million monthly users, 242 countries) built a real-time classroom analytics system on ClickHouse Cloud that ingests roughly 8 billion events per month and returns query results at 45ms median and 690ms P99 — during live lessons, not batch jobs. The team of 65 has no dedicated data team, and they went from setup to production in about a month. They leaned on MergeTree and AggregatingMergeTree engines, materialized views, and HyperLogLog for unique visitor counts. The operational overhead point is worth noting: 14 requests per second with minimal ops investment is a reasonable data point for teams evaluating ClickHouse Cloud against self-managed alternatives.</p>
<hr>
<h2>Also notable</h2>
<ul>
<li><strong>Billion-scale vector search in Elasticsearch/OpenSearch:</strong> A conference talk covers quantization (scalar, product, binary) reducing memory from ~4TB to hundreds of GB at billion-vector scale, GPU-accelerated indexing with CAGRA, and why quantization with reranking is the practical path forward.</li>
<li><strong>Privacy in real-time pipelines:</strong> A talk using Netflix, AOL, Strava, and NYC taxi datasets to show that stripping obvious identifiers isn’t enough; covers tokenization vs. hashing, k-anonymity, bucketing, and why privacy controls need to be applied at ingestion, not downstream.</li>
<li><strong>JPA/Hibernate N+1 and connection pool pitfalls:</strong> A conference talk quantifies connection pooling as roughly 4,000x faster than direct connections, flags Spring’s open-session-in-view default as a common source of unexpected behavior, and covers DTO projections via JPQL constructor expressions as the right pattern for read-only queries.</li>
<li><strong>ClickHouse MCP and agentic data stack:</strong> ClickHouse published its strategy for AI agents as database users, including open-source and remote MCP servers, chDB 4 with a Pandas-like API, and agent-friendly docs via llms.txt. OpenAI, Anthropic, and <a href="http://Character.AI" target="_blank" rel="noopener nofollow">Character.AI</a> are cited as external adopters at petabyte scale.</li>
<li><strong>Snowflake Openflow series wrap-up:</strong> The final installment compares dbt, Dynamic Tables, native SQL, and Snowpark for transformation workloads, and maps ingestion patterns (bulk, streaming, CDC, external tables) to latency and governance requirements.</li>
<li><strong>AWS Lambda on Managed Instances:</strong> Lambda now supports 32 GB memory and 16 vCPUs on Lambda Managed Instances, with a 4x increase to the file descriptor limit — relevant for workloads that were previously hitting memory or concurrency ceilings.</li>
<li><strong>Aurora PostgreSQL joins AWS Free Tier:</strong> Aurora PostgreSQL is now on the AWS Free Tier, and the express configuration option enables near-instant database creation — useful for dev/test environments that previously defaulted to RDS.</li>
<li><strong>Amazon OpenSearch Cluster Insights in console:</strong> Cluster Insights health recommendations are now visible directly in the AWS Management Console and can be published to EventBridge for automated alerting, available at no extra cost on OpenSearch 2.17+.</li>
<li><strong>sqlite-vec semantic search in Rails:</strong> A practical walkthrough of adding kNN search to a Rails app using sqlite-vec, a self-hosted EmbeddingGemma-300m embedding service in Docker, and Kamal for deployment — complementing existing FTS5 full-text search with RRF hybrid fusion planned next.</li>
<li><strong>Multi-cloud Redis provisioning with OpenTofu:</strong> The zopdev/opentofu-modules redis/ module provisions ElastiCache, Memorystore, and Azure Cache from a unified interface, handling cluster mode selection, multi-AZ failover, encryption at rest, and Kubernetes ExternalName service creation.</li>
<li><strong>Knex.js status update:</strong> The JavaScript SQL query builder for PostgreSQL, CockroachDB, MySQL, SQLite, and Redshift published a post on its current state and roadmap.</li>
</ul>


Query optimization, schema design, and data store coverage across PostgreSQL, MongoDB, Redis, and more.

Databases Digest

A curated roundup of database and data engineering news. Apache Iceberg V3 introduces Deletion Vectors that reduce write amplification by up to 4,358x versus rewriting Parquet files, plus Snowflake adds bidirectional interoperability with external catalogs like AWS Glue and Unity Catalog. SQL Server 2025 ships six native T-SQL regex functions including regexp_split_to_table and regexp_matches, replacing clunky CHARINDEX/PATINDEX workarounds. Debezium 3.4.3 patches a silent data corruption bug in the PostgreSQL pgoutput plugin triggered by multibyte UTF-8 characters in table/column names. Padlet processes 8 billion events per month through ClickHouse Cloud with 45ms median query latency using MergeTree engines, materialized views, and HyperLogLog. Also covered: billion-scale vector search in Elasticsearch, JPA/Hibernate N+1 pitfalls, AWS Lambda Managed Instances with 32GB memory, Aurora PostgreSQL on AWS Free Tier, and sqlite-vec semantic search in Rails.

Iceberg V3 rewrites the delete path, SQL Server 2025 gets native regex