<p><strong>TLDR:</strong> Pinterest Engineering published a detailed breakdown of how they eliminated most Spark out-of-memory failures across tens of thousands of daily jobs using auto memory retries and better observability. A separate post shows a Snowflake team replacing a polling anti-pattern with a fully native event chain, saving 240 compute credits a month. Dremio shipped full Apache Iceberg V3 support in Dremio Cloud, adding deletion vectors and a VARIANT type for JSON. Popsink released a connector for replicating IBM i DB2 into Snowflake via CDC, which is notable because AS/400 journaling is incompatible with most standard replication tooling.</p>
<hr>
<h2>Pinterest’s 96% reduction in Spark OOM failures</h2>
<p>Pinterest tackled Spark OOM failures at scale with three things working together: executor-level memory metrics to find skewed partitions, Spark config tuning including adaptive query execution, and Auto Memory Retries that automatically restart failed jobs with higher memory allocations. The rollout went staged from ad hoc to scheduled jobs, with dashboards tracking recovered jobs and cost impact. Two things worth noting from their write-up: scheduler performance degraded on large TaskSets, and Apache Gluten compatibility required special handling. Their next target is proactive memory increases for high-risk stages before they fail, which is the right direction.</p>
<h2>Replacing Snowflake polling with a native event chain</h2>
<p>A data team was burning 240 compute credits per month polling a logging table every 10 minutes to detect job completion. The fix is clean: source system drops a CSV to S3, Snowpipe ingests it, a Snowflake Stream detects the new row, a Task fires the ETL procedure. No external orchestrator, no wasted warehouse warm-ups. This is a good pattern to have in your back pocket if you’re running scheduled Snowflake workloads that depend on upstream signals.</p>
<h2>Dremio adds Apache Iceberg V3 support</h2>
<p>Dremio Cloud now supports Iceberg V3, which brings the VARIANT type for semi-structured JSON, deletion vectors for faster CDC, and improved schema evolution. The open catalog runs on Apache Polaris and supports interoperability with Spark, Flink, Trino, and DuckDB. Dremio also co-created Apache Arrow and Apache Polaris, so this isn’t just a feature announcement — they have real skin in the Iceberg ecosystem.</p>
<h2>IBM i DB2 to Snowflake via CDC</h2>
<p>Popsink released a Snowflake Native Application that replicates from IBM i (AS/400) DB2 into Snowflake using CDC. The hard part here is that IBM i uses proprietary journaling instead of WAL or binary logs, which breaks most generic replication tools. The connector reads those journals directly, runs entirely inside your Snowflake account using External Access Integration, and supports sync intervals down to 20 seconds. If you’re dealing with AS/400 data, this is worth a look.</p>
<hr>
<h2>Also notable</h2>
<ul>
<li><strong>pGenie for PostgreSQL:</strong> SQL-first tooling that validates queries, manages indexes, and generates type-safe client SDKs from plain SQL migrations — catches sequential scans and nullability issues at generation time, not runtime.</li>
<li><strong>Snowflake AI_CLASSIFY:</strong> Practical guide to using Snowflake’s Cortex AI_CLASSIFY function for text classification directly in SQL, including few-shot examples and building dynamic category arrays with ARRAY_AGG and CTEs.</li>
<li><strong>Vercel dashboard SQL:</strong> Vercel added Query, Data Editor, and Schema tabs to marketplace database integrations, supporting AWS Aurora Postgres, Neon, Prisma, and Supabase — useful for quick inspection without leaving the dashboard.</li>
<li><strong>PlanetScale traffic control patterns:</strong> Walkthrough of Go patterns for attaching resource budgets to Postgres traffic using SQLCommenter tags, covering per-service isolation, feature flag tagging, and graceful degradation on SQLSTATE 53000.</li>
<li><strong>TigerFS:</strong> Experimental filesystem that mounts a PostgreSQL database as a directory, letting you use ls, cat, find, and grep against database-stored files instead of APIs.</li>
<li><strong>Appwrite 1.9.0:</strong> Adds MongoDB as an alternative backend to MariaDB for self-hosted deployments — the choice is permanent at install time via a new web wizard.</li>
<li><strong>NoSQL migration ID generation:</strong> InfoQ piece on building a sequence service backed by DynamoDB with a 2-tier caching architecture to handle distributed unique ID assignment after moving off a relational database.</li>
<li><strong>Free PostgreSQL providers 2026:</strong> Curated list of 10 free-tier Postgres hosts including Supabase, Neon, Railway, Render, and Aiven, with free tier limits and ideal use cases for each.</li>
</ul>


Query optimization, schema design, and data store coverage across PostgreSQL, MongoDB, Redis, and more.

Databases Digest

A curated roundup of data engineering news and patterns. Pinterest reduced Spark out-of-memory failures by 96% using auto memory retries, adaptive query execution, and executor-level memory observability across tens of thousands of daily jobs. A Snowflake team eliminated a polling anti-pattern by replacing it with a native event chain using Snowpipe, Streams, and Tasks, saving 240 compute credits monthly. Dremio Cloud shipped Apache Iceberg V3 support including deletion vectors, a VARIANT type for JSON, and interoperability with Spark, Flink, Trino, and DuckDB via Apache Polaris. Popsink released a CDC connector for replicating IBM i DB2 (AS/400) into Snowflake, handling proprietary journaling that breaks most standard replication tools. Also covered: PostgreSQL tooling (pGenie, TigerFS), Snowflake AI_CLASSIFY for SQL-based text classification, Vercel dashboard SQL tabs, PlanetScale traffic control patterns in Go, Appwrite 1.9.0 with MongoDB support, DynamoDB-backed sequence services, and a list of free PostgreSQL providers.

Pinterest cuts Spark OOM failures by 96%, Snowflake gets event-driven ETL without an orchestrator