Welcome back to Data This Week!
This week is heavy on internals and architectural trade-offs. We’re busting myths around S3 Tables compaction pricing, going deep on how DuckDB’s vectorized engine actually works, stress-testing Snowflake’s managed Iceberg against 13 known OSS pain points, and catching a fascinating query rewrite trick both Snowflake and Databricks pull silently on your NOT EXISTS clauses. On the infrastructure side, we revisit the case for partition affinity over a Redis cache and explore Supabase Pipelines’ new automatic schema replication.
Here are the top reads, tools, and community discussions for senior data folks this week.
📚 Blogs to Read
The Hidden Truths of Amazon S3 Tables
Real-world testing of S3 Tables debunks several viral complaints that have been circulating since the product launched. The most important correction: AWS recently slashed compaction pricing by 90%, making the “prohibitively expensive” narrative stale. However, the testing reveals a genuine gotcha — compaction runs on a rigid, unconfigurable ~3-hour internal schedule. If you have strict streaming freshness SLAs, this hidden schedule will absolutely bite you. For small-to-medium velocity workloads, though, S3 Tables is significantly cheaper and simpler than the internet currently claims. Worth re-evaluating your assumptions before dismissing it.
DuckDB Internals Part 2: Vectorized Execution
A thorough deep dive into why DuckDB is as fast as it is, by examining its departure from the traditional Volcano (iterator) execution model in favor of DataChunks. DuckDB processes data in batches of ~2,048 rows using four distinct vector formats — Flat, Constant, Dictionary, and Sequence — each chosen to minimize memory allocation for the data shape at hand. This keeps the working set small enough to fit in the CPU cache while eliminating the per-row function call overhead that heavily bottlenecks traditional row-at-a-time engines. If you’re optimizing DuckDB queries or building on top of its API, understanding these internals is essential.
Running Iceberg at Scale: Stress-Testing All 13 Iceberg Challenges on Snowflake
A 10.8-million-row financial lakehouse test puts Snowflake’s managed Iceberg implementation through the full gauntlet of known OSS Iceberg operational challenges. The headline finding: Snowflake’s engine automatically handles small-file accumulation and manifest compaction in the background for pennies. More impressively, it merges deletion vectors at the storage engine level, virtually eliminating the read-latency penalties that normally plague Merge-on-Read (MOR) architectures. If you are running open-source Iceberg on Spark or Trino and absorbing the full platform engineering overhead yourself, this is a worthwhile benchmark to read before your next infrastructure review.
Snowflake & Databricks Both Didn’t Execute Your NOT EXISTS Clause
Neither Snowflake nor Databricks actually execute a correlated NOT EXISTS subquery the way you wrote it. Both engines automatically rewrite it into a distributed LEFT ANTI JOIN, because row-by-row subquery evaluation is fundamentally incompatible with a distributed execution model. Snowflake goes one step further by injecting an aggregate DISTINCT on the join keys from the right-hand table before the join, minimizing the number of rows passed to the join operator. A must-read if you’re writing complex SQL for distributed systems and want to understand the query plans your engine actually executes.
You Don’t Need a Cache Between Your Consumer and Your Database — You Need Partition Affinity
Before defaulting to Redis or Memcached to protect your database from read-heavy workloads, consider routing requests for the same data partition consistently to the same consumer instance. Implementing partition affinity naturally improves CPU cache locality at the application layer, reducing the raw connection load on the database without requiring an entirely new infrastructure component. This piece makes a compelling argument that the reflex to add a distributed cache is often an expensive, complexity-compounding solution to a problem that routing policy can solve more elegantly.
Automate Creating AWS Glue Data Catalog Views with the AWS SDK
The AWS SDK now supports the programmatic creation of multi-dialect Glue Data Catalog views — defining both ATHENA and SPARK dialects simultaneously in a single API call. For data mesh architectures, this is a meaningful unlock: producer accounts can now fully automate their CI/CD pipelines using cross-account IAM definer roles, while centralized governance in the master account remains intact. If cross-account data access management is a bottleneck in your organization today, this is worth a close read.
Supabase Pipelines Is Now in Public Alpha
Supabase’s Rust-based CDC pipeline now automatically replicates Postgres schema changes downstream to data warehouses. This addresses one of the most painful failure modes of logical replication: someone drops or renames a column in production and silently breaks the analytics pipeline. Supabase Pipelines detects column additions, removals, and renames in Postgres and safely applies them to BigQuery. Snowflake, DuckLake, and ClickHouse support are on the immediate roadmap. Worth watching closely if you’re building automated data flows from Postgres.
🛠️ Tools
Lightstream
What it is: A new Rust-based, high-performance data transport library with Python bindings, released just this week. Lightstream moves Apache Arrow, Protobuf, and MessagePack data across processes, IPC, sockets, and networks without the overhead of a gRPC stack. Its standout feature is native implementation of the Arrow PyCapsule protocol, which means libraries like DuckDB, Polars, and PyArrow can consume data straight off the wire without an intermediate conversion step. Early open benchmarks on 50 Gbps EC2 networks show it outperforming Apache Arrow Flight on every measured axis.
💬 Community Sentiments
Is Databricks and Fabric Overtaking Snowflake?
A massive thread on r/dataengineering tackled the current state of the platform wars, and the consensus among senior practitioners is nuanced. Snowflake and Databricks remain the undisputed top tier, with their head-to-head competition driving exceptional R&D — Snowflake successfully smoothing out ML workloads while Databricks continues strengthening its data warehousing capabilities. Microsoft Fabric, however, is drawing heavy fire from engineers on the ground: capacity throttling, out-of-sync workspaces, and surprisingly steep compute costs for relatively small workloads are recurring complaints, despite its aggressive bundling with existing Microsoft/PowerBI contracts. The general takeaway: choose Databricks or Snowflake based on your team’s Python-vs-SQL strengths, but tread very carefully before migrating enterprise workloads to Fabric.
That’s all for this week! See you in the next edition.