Welcome back to Data This Week!
This week’s lineup spans the full breadth of the modern data stack—from rethinking what Databricks actually is, to pluggable storage engines reshaping PostgreSQL, to the landmark Iceberg 1.11.0 release. We also cover some highly practical operational topics: egress cost management, automated JDBC caching, and graph infrastructure at Airbnb scale. A strong week for both architectural thinking and hands-on engineering.
Here are the top reads, tools, and community discussions for senior data folks this week.
📚 Blogs to Read
Understanding Databricks: What It Is and Why It Matters
It’s easy to view Databricks as just a managed Spark service, but this piece breaks down why that mindset leaves 80% of the platform on the table. It provides a great architectural overview of how the platform has evolved into a unified workspace encompassing Delta Lake, declarative pipelines via Lakeflow, Mosaic AI, and Unity Catalog for governance. A solid read for folks looking to move beyond simple PySpark notebooks. Read more →
A Field Guide to Alternative Storage Engines for PostgreSQL
Postgres 12 shipped the Table Access Method (TAM) API in 2019, and the community is finally delivering on the promise of pluggable storage. This article explores the current state of alternative engines, from heap replacements like OrioleDB (which uses undo-logs to eliminate MVCC bloat) to columnar TAMs like Citus and Hydra. It’s a fascinating look at the architectural shifts turning Postgres into a truly hybrid analytical and transactional powerhouse. Read more →
An In-Depth Overview of the Apache Iceberg 1.11.0 Release
The 1.11.0 release is a massive structural leap for Iceberg. Alex Merced breaks down the critical PRs, highlighting major security wins like native manifest list encryption to protect table metadata structures. On the performance side, the release introduces a pluggable File Format API—decoupling the core engine to support next-gen formats like Vortex and Lance—and adds deletion vector pruning during snapshot validation to drastically reduce query planning overhead for massive tables. For those of us running continuous Spark or Flink ingestion, the new scheduled credential refresh and Flink post-commit maintenance (running compaction directly from the sink) are absolute lifesavers. Read more →
Egress Problems and Where to Find Them (PlanetScale)
A highly practical breakdown of database egress costs. Egress (data transferred out over the public internet) can silently inflate your cloud bill if you aren’t careful. The article details how fetching too much data (SELECT *), failing to paginate, and aggressively returning full objects on INSERT operations are the usual culprits. It’s a great reminder to lean on specific column selections, caching layers, and private links to keep applications fast and cheap.
Read more →
Automated JDBC Query Caching with the AWS Advanced JDBC Wrapper
AWS just announced the Remote Query Cache Plugin for their Advanced JDBC Wrapper. You can now automatically cache query results in Amazon ElastiCache for Valkey simply by adding a SQL hint like /* CACHE_PARAM(ttl=300s) */ to your SELECT statements. The plugin transparently intercepts the query, checks the cache, and serves it in microseconds. It’s a massive win for read-heavy applications looking to reduce database load without a major architectural rewrite.
Read more →
Scaling Airbnb’s Identity Graph with a Unified Knowledge Graph Infrastructure
Airbnb transitioned their massive Identity Graph (7 billion nodes, 11 billion edges) from a legacy relational database and key-value store setup to a unified, paved-path infrastructure. They utilized a heavily optimized internal fork of JanusGraph backed by DynamoDB. Tackling a growth rate of 5 million new edges per day, they implemented custom transaction strategies and parallel query execution to hit strict latency requirements for their Trust and Safety use cases. Read more →
🛠️ Tools
OrionBelt Semantic Layer
What it is: An API-first semantic engine that compiles declarative YAML models into optimized SQL for BigQuery, Postgres, Snowflake, DuckDB, and Databricks. It provides “Analytics as Code,” allowing both traditional applications and AI assistants—via its Claude Desktop MCP server—to query data using actual business concepts rather than raw table schemas.
💬 Community Sentiments
dbt-colibri v0.3.4: Local Column-Level Lineage
There’s a lot of excitement over on r/dataengineering regarding the recent release of dbt-colibri v0.3.4. Anyone who has managed a sprawling dbt project knows that assessing the impact of a downstream change using the default model-level lineage graph is an absolute nightmare. Colibri brings local column-level lineage that successfully tracks columns through complex WHERE and JOIN clauses. It now supports a wide range of adapters (Snowflake, BigQuery, Databricks, DuckDB) and even tracks ephemeral models. The community is praising it for making impact analysis incredibly intuitive while keeping it accessible right in your local environment.
Read more →
That’s all for this week! See you in the next edition.