Welcome back to Data This Week!
This week’s lineup dives deep into the operational mechanics of the modern data stack. We are seeing a massive push toward blurring the lines between operational databases and data lakes—highlighted by a new Postgres extension that natively tiers data directly to Iceberg without ETL. We also look at Trino’s latest SQL standard upgrades, the architectural realities of scaling Postgres connection pools, and why your incremental data models might be silently failing.
Here are the top reads, tools, and community discussions for senior data folks this week.
📚 Blogs to Read
PostgreSQL 19: Controlling the Planner with pg_plan_advice
A new module in Postgres 19 lets you hint or force query execution plans using a mini-language (like JOIN_ORDER and INDEX_SCAN). While it operates by constraining the planner rather than replacing it, this is a powerful tool for stabilizing plans that the engine might otherwise optimize poorly due to data distribution changes. Just be careful—overriding the planner’s judgment without understanding the mechanics can easily backfire.
Read more →
Announcing Flink’s Native S3 FileSystem
For years, S3 support in Flink meant choosing between two Hadoop-based plugins, each bringing massive CVE overhead and feature compromises. Flink 2.3 introduces a ground-up, Hadoop-free S3 filesystem (flink-s3-fs-native). It drops the JAR size from ~90MB to 13MB, uses AWS SDK v2 for async I/O, and delivers up to 2x faster checkpoints while fully supporting exactly-once sinks.
Read more →
Why Incremental Models Pass Tests and Still Go Wrong
Incremental models aren’t just faster tables; they are tables with memory. This piece breaks down why relying on simple max(timestamp) cutoffs is a trap in the real world of late-arriving data, mutable sources, and complex window functions. It’s a great reminder that the incremental transition rule is the model, not just an optimization detail, and requires rigorous testing against a full recomputation.
Read more →
Trino’s Summer of Grammar
Trino 482 closes several gaps in ISO standard SQL support. Highlights include BETWEEN SYMMETRIC (bounds order doesn’t matter), explicit three-valued logic (IS [NOT] TRUE/FALSE/UNKNOWN), and UNIQUE / MATCH predicates for subqueries. Note the breaking change: char now coerces to varchar (dropping trailing padding) instead of the other way around, finally fixing a long-standing source of comparison bugs.
Read more →
How PgBouncer Works (And Why Postgres Needs It)
A deep dive into the mechanics of Postgres connection overhead—where every connection forks a 1.5–15MB process. The post breaks down the critical differences between session, transaction, and statement pooling. If you’ve ever wondered why SET commands, prepared statements, or temp tables mysteriously break when routing through port 6432, this is the definitive explanation of transaction pooling’s leaky abstraction.
Read more →
Turning Scattered Data into Queryable Segments at Scale
Razorpay details their architectural journey in building a highly scalable Customer Data Platform (CDP). Dealing with massive volumes of scattered transactional and event data requires a robust ingestion and transformation layer to create queryable, unified segments for downstream analytics and targeted marketing. Read more →
Multi-Region Identity-Based Access to Amazon Redshift and S3 Tables
For enterprise data platforms distributed across AWS regions, managing access roles is a headache. This AWS guide demonstrates how to use IAM Identity Center Multi-Region alongside Trusted Identity Propagation and Lake Formation. It allows a single identity to seamlessly authenticate and inherit fine-grained access to Redshift and S3, without juggling complex cross-account IAM role configurations. Read more →
🛠️ Tools
pgEdge ColdFront
What it is: A newly open-sourced, Postgres-native extension that transparently tiers data between hot Postgres partitions and cold Apache Iceberg storage (Parquet on S3/Azure/GCS). It eliminates the need for separate ETL pipelines—your application queries a single table using standard SQL, and the cold tier remains fully writable (supporting UPDATE and DELETE). Perfect for heavily regulated environments with long retention mandates.
💬 Community Sentiments
The Right Way to Use Databricks Professionally
Over on r/dataengineering, a new hire expressed frustration over their team exclusively using the Databricks UI/notebooks and oversized clusters for simple Pandas scraping jobs. The community consensus was clear: production Databricks requires treating it like standard software engineering. The recommended stack is Databricks Asset Bundles (DABs) for CI/CD, Databricks Connect for local VS Code development, and swapping plain Pandas for pyspark.pandas (or utilizing single-node/serverless compute) to actually leverage the underlying cluster efficiency.
That’s all for this week! See you in the next edition.