Welcome back to Data This Week!
This week’s lineup covers major shifts in storage paradigms and operational scaling. From Databricks fundamentally rethinking the database storage layer to eliminate CDC, to CRED’s masterclass in orchestrating 120+ database upgrades without dropping a single event, the focus is heavily on doing more with less infrastructure. We also look at how Cloudflare built a unified lakehouse to handle billions of events and how AWS is pushing business context closer to the data layer.
Here are the top reads, tools, and community discussions for senior data folks this week.
📚 Blogs to Read
Amazon S3 Adds Annotations for Business Context
AWS just added the ability to attach up to 1 GB of custom JSON, XML, or YAML metadata directly to S3 objects. Instead of building separate metadata stores, you can now embed business context directly onto the object. When synced with S3 Metadata (which creates managed Iceberg tables), AI agents and analytics tools can query this context natively to discover and understand your data. Read more →
How We Built Cloudflare’s Data Platform and an AI Agent on Top of It
Cloudflare processes over a billion events per second. To tame their data sprawl, they built “Town Lake,” an internal lakehouse powered by Apache Trino, Iceberg, and R2. The highlight is their “default-closed” governance model with automated PII scanning. On top of this, they built Skipper, an AI data agent that writes accurate SQL by leveraging four distinct layers of metadata context—solving the classic LLM hallucination problem for enterprise data. Read more →
Lakebase & LTAP: Rethinking Database Storage
Databricks is proposing LTAP (Lakehouse Transaction Analytical Processing) as an evolution of HTAP. Instead of trying to unify workloads in a single engine, their Lakebase Postgres offering externalizes the database storage entirely. The write-ahead log (WAL) and data files live as Parquet/Iceberg tables in object storage. This means the analytical warehouse runs on the exact same fresh data that transactions just wrote—effectively eliminating the need for CDC pipelines. Read more →
Bringing Iceberg Native Views to Apache Hive
For teams moving to Iceberg REST catalogs, the lack of a Hive Metastore (HMS) creates a gap for storing logical views. A new Hive update (HIVE-29578) solves this by supporting native Iceberg logical views. The view definitions are now stored directly in the Iceberg catalog alongside tables, bringing fully decoupled metastores and cross-engine view interoperability (Spark, Flink, Trino) to the Hive ecosystem. Read more →
What Breaks Your Snowflake Pipeline at 100M+ Rows
Pipelines that run perfectly at 5M rows often grind to a halt at 100M. The most common culprit? Naive dbt incremental models doing full table scans during MERGE operations. This piece breaks down how to fix it by leveraging Snowflake’s micro-partition pruning—specifically, ensuring your target table is clustered on a date column and pushing that date predicate directly into the merge condition.
Read more →
Data Architecture Isn’t One-Size-Fits-All
A solid practitioner’s guide cutting through the hype of architectural patterns. It compares the Lambda architecture (dual batch/speed layers), the Kappa architecture (streaming-first), and the Medallion architecture (Bronze/Silver/Gold logical phases). A great refresher for senior engineers on matching the right pattern to actual business latency requirements rather than adopting the newest trend. Read more →
How CRED Uses Amazon RDS Blue/Green Deployments at Scale
Upgrading 120 production databases usually means weekend war rooms. CRED automated this away using RDS Blue/Green deployments, achieving 2-minute switchovers with zero data loss. The real alpha in this post is their approach to CDC pipelines: they automated AWS DMS and Debezium checkpoint tracking to seamlessly pause on the blue environment and resume on the green—without duplicating events. Read more →
Deploying Multi-Pipeline Databricks Bundles
Managing a real data product means orchestrating multiple pipelines, jobs, and CI/CD flows simultaneously. This deep dive into Databricks Asset Bundles (DABs) shows how to deploy a complete project as a single coherent unit. The key takeaway: use bundle cross-reference syntax (${resources.pipelines.*.id}) to wire pipeline IDs to workflows dynamically—completely eliminating manual UUID pasting and mismatched environments.
Read more →
🛠️ Tools
Hardwood
What it is: A blazing-fast, lightweight Apache Parquet reader built specifically for the JVM (version 1.0 recently released). If you’re running Java or Scala data engineering stacks and want to read Parquet files natively without pulling in heavy Spark or Hadoop dependencies, this zero-fluff reader is a massive operational win.
💬 Community Sentiments
Technical Article on LTAP (Databricks)
Over on r/dataengineering, the community is actively debating the Databricks Lakebase/LTAP architecture. The general consensus is cautiously optimistic. Practitioners are pointing out that this fundamentally shifts the Medallion architecture—if your operational Postgres database uses Iceberg on S3 as its source of truth, the entire Bronze ingestion and CDC pipelining layer is effectively eliminated. However, some healthy skepticism remains around how cost-to-serve and latency will hold up in production compared to traditional read replicas.
That’s all for this week! See you in the next edition.