<p><strong>TLDR:</strong> Databricks pushed Apache Iceberg v3 into public preview with row lineage, deletion vectors, and a native VARIANT type. PostgreSQL 19 is adding three SQL-callable DDL functions that expose what pg_dump produces as queryable output. PlanetScale shipped smarter query tagging in Postgres Query Insights with cardinality controls baked in. Redpanda Connect now has a native Oracle CDC connector that cuts Debezium and the JVM entirely.</p>
<hr>
<h2>Apache Iceberg v3 enters public preview on Databricks</h2>
<p>Databricks Runtime 18.0+ with Unity Catalog enabled gets three meaningful additions: Row Lineage (permanent row IDs and sequence numbers for change tracking), Deletion Vectors (lightweight delete files claimed to be up to 10x faster than copy-on-write), and a VARIANT type for semi-structured data queryable with standard SQL. The historical tradeoff between performance and cross-engine interoperability was the main knock on Iceberg at scale, and these features are a direct answer to that. Delta Lake with UniForm still lets you write once and read as Iceberg from Snowflake, BigQuery, Redshift, and Athena. Databricks engineers are already driving v4 proposals including an adaptive metadata tree and a modernized statistics model.</p>
<h2>PostgreSQL 19 adds pg_get_database_ddl, pg_get_role_ddl, pg_get_tablespace_ddl</h2>
<p>Three new functions land in PostgreSQL 19 that return the DDL needed to recreate databases, roles, and tablespaces — essentially surfacing what pg_dump produces as callable SQL. Each accepts optional parameters like <code>pretty</code> for formatted output and <code>owner</code> to include ownership info. Worth noting: <code>pg_get_role_ddl()</code> intentionally omits password information. Practical use cases include catalog-driven schema auditing and scripting migrations without shelling out to pg_dump.</p>
<h2>PlanetScale adds tag-aware aggregation to Postgres Query Insights</h2>
<p>Query tags (key-value pairs embedded via sqlcommenter) now flow into aggregated query data in PlanetScale’s Postgres Query Insights, not just individual notable query logs. The interesting engineering detail is how they handled cardinality: after 20 unique values per tag key per query pattern, the tag collapses to a wildcard. When unique tag combinations exceed 50 per 15-second interval, the highest-cardinality tag gets dynamically collapsed. Without those controls, high-cardinality tags like <code>request_id</code> would blow up the aggregation pipeline. The result is per-tag database statistics and tag distribution across query patterns without the storage explosion.</p>
<h2>Redpanda Connect gets native Oracle CDC, no JVM required</h2>
<p>Redpanda Connect v4.83.0 ships an <code>oracledb_cdc</code> connector that reads from Oracle LogMiner and streams inserts, updates, and deletes as structured events. No Debezium, no Kafka Connect cluster, no external offset storage, no JVM — the whole thing runs as a single Go binary with a YAML config. It handles initial snapshots, transitions to streaming from redo logs, supports regex-based table filtering, and tracks schema changes via <code>ALL_TAB_COLUMNS</code>. Schema Registry integration for Avro encoding and Oracle Wallet support for regulated environments are included.</p>
<hr>
<h2>Also notable</h2>
<ul>
<li><strong>Uber Hive federation:</strong> Uber migrated 16,000+ datasets totaling 10+ PB from a monolithic Hive instance to a federated, domain-specific architecture using pointer-based Metastore redirects — zero downtime, and over 1 PB of HDFS space reclaimed from stale datasets.</li>
<li><strong>Iceberg vs Delta Lake:</strong> A detailed comparison concludes the feature gap has narrowed enough that ecosystem fit is now the primary decision factor — Iceberg for multi-engine vendor-neutral stacks, Delta Lake for Databricks-centric shops where Change Data Feed and multi-table transactions matter.</li>
<li><strong>MSBuild.Sdk.SqlProj v4.0.0:</strong> The SDK-style .NET project format for SQL Server schemas now supports <code>dotnet publish /t:PublishContainer</code>, bundling SqlPackage and the .dacpac into a self-contained Docker image for reproducible, tool-free database deployments.</li>
<li><strong>Snowpipe Streaming error logging:</strong> Snowflake’s error logging feature now integrates with the Snowpipe Streaming high-performance architecture — enable with <code>ERROR_LOGGING = TRUE</code> on a table and failed rows land in a SQL-queryable dead-letter table with full payload and metadata, retained up to 14 days at standard storage cost.</li>
<li><strong>ClickHouse clusters and replication:</strong> A solid reference post clarifying that ClickHouse cluster topology (routing) and replication are independent layers — key guidance includes using <code>{uuid}</code> macros in <code>default_replica_path</code>, always setting <code>internal_replication = true</code> with Distributed tables, and keeping cluster configs identical across all nodes.</li>
<li><strong>Zalando on Databricks Unity Catalog:</strong> Zalando’s unified data foundation for 50M+ customers uses a dual-catalog pattern with Dynamic Views, a “Metric as Code” semantic layer via Databricks Metric Views, and Databricks Genie for SQL-free analytics — time-to-insight for performance meetings dropped from hours to minutes.</li>
<li><strong>Instant 1.0 backend architecture:</strong> A backend-as-a-service for AI-coded apps using a client-side triple store with Datalog, a Clojure WAL-based invalidator, and a single Postgres triples table in EAV pattern augmented with count-min sketches for query planning.</li>
<li><strong>Soft delete unique indexes:</strong> PostgreSQL handles this cleanly with partial indexes using a WHERE clause; MySQL requires a generated column returning 1 for active rows and NULL for deleted ones, exploiting MySQL’s allowance of multiple NULLs in a unique index.</li>
<li><strong>Amazon Timestream for InfluxDB:</strong> Now supports customer-defined weekly maintenance windows with IANA timezone identifiers and automatic DST handling, configurable via console, CLI, or SDK.</li>
<li><strong>Snowflake Cortex Code:</strong> Snowflake’s native AI coding assistant integrates into VS Code via CLI, understands your schemas and metadata, and can generate SQL, Snowpark Python, dbt models, and Streamlit apps from natural language — supports Llama, Mistral, and Claude as backing models.</li>
</ul>


Query optimization, schema design, and data store coverage across PostgreSQL, MongoDB, Redis, and more.

Databases Digest

A roundup of database and data engineering news: Apache Iceberg v3 enters public preview on Databricks Runtime 18.0+ with row lineage, deletion vectors, and a VARIANT type. PostgreSQL 19 adds three SQL-callable DDL functions (pg_get_database_ddl, pg_get_role_ddl, pg_get_tablespace_ddl) that surface pg_dump output programmatically. PlanetScale ships tag-aware aggregation in Postgres Query Insights with cardinality controls to prevent storage explosion. Redpanda Connect v4.83.0 adds a native Oracle CDC connector via a single Go binary, eliminating Debezium and the JVM. Also covered: Uber's 10+ PB Hive federation migration, Iceberg vs Delta Lake comparison, ClickHouse replication guidance, Snowflake Cortex Code AI assistant, and soft delete unique index patterns across PostgreSQL and MySQL.

Apache Iceberg v3 hits Databricks preview, PostgreSQL 19 gets DDL functions