Welcome back to Data This Week!
This week’s lineup covers major leaps in data lake operations and real-time ingestion. From Amazon MSK eliminating the Kafka Connect tax for Iceberg to Iceberg v3 introducing native row-level lineage for true CDC, the focus is heavily on streamlining data infrastructure. We also look at Spotify’s approach to serving online point queries directly from the data lake, and a blazing-fast Rust terminal tool for your databases.
Here are the top reads, tools, and community discussions for senior data folks this week.
📚 Blogs to Read
Deliver Apache Kafka Data to Streaming Tables for Apache Iceberg
Amazon MSK Express brokers just launched direct delivery to Apache Iceberg streaming tables on S3. This eliminates the need to deploy, scale, or maintain Kafka connectors, Flink jobs, or custom consumers just to land streaming data. It also performs intelligent inline compaction during ingestion, solving the notorious “small-file problem” without writing a single line of code. A massive operational win for real-time data architectures.
Apache Iceberg Row Lineage: Tracking Data Lineage at the Row Level
Iceberg v3 brings a highly anticipated feature: native row-level tracking. Instead of relying on file-level snapshot diffs or unreliable updated_at timestamps, every row now carries a stable _row_id and a _last_updated_sequence_number. These hidden metadata columns allow you to pull exact row-level changes with a simple WHERE clause, making CDC and auditing directly on the data lake vastly simpler.
Indexing the Data Lake for Online Point Queries
Spotify engineering shares a deep dive into how they are blurring the lines between operational databases and analytical data lakes. They detail their architectural approach to building a custom indexing layer over their data lake, enabling high-concurrency, low-latency point queries without needing to reverse-ETL everything into a dedicated key-value store.
A Deep Dive into File Compression: How Data Gets Smaller
Stop blindly defaulting to GZIP. This deep dive breaks down the mechanics of file compression codecs — Snappy, Zstd, LZ4, and others — explaining how these algorithms actually compress data under the hood. More importantly, it provides a senior-level framework for choosing the right codec to balance storage footprint against compute overhead in modern data pipelines.
Every Way to Get Data into Snowflake & How to Pick the Right One
An exhaustive architectural breakdown of Snowflake ingestion patterns. The piece evaluates Snowpipe, COPY INTO, Snowpipe Streaming, and external stages, offering clear decision matrices on when to use each based on latency requirements, volume, and cost constraints.
Are My Fabric Tables Healthy?
For teams adopting Microsoft Fabric, maintaining underlying table health is critical. This guide covers the practical queries and maintenance strategies required to monitor file fragmentation, optimize performance, and keep your Fabric lakehouse running efficiently at scale.
Interesting Links: July 2026
Robin Moffatt’s monthly roundup is always worth a scan. He curates some of the best talks, releases, and niche technical write-ups from across the data engineering landscape over the past month.
🛠️ Tools
Rainfrog
What it is: A blazing-fast, lightweight database GUI for the terminal, written in Rust. It supports Postgres, MySQL, SQLite, and DuckDB — along with anything using the Postgres wire protocol, such as Redshift. Featuring Vim-like keybindings, keyword highlighting, and quick data filtering, it’s the perfect refuge for engineers who want to query their databases without ever leaving the terminal.
💬 Community Sentiments
How to Efficiently Load Large Data into Iceberg
Over on r/dataengineering, a highly technical discussion is unfolding around the bottlenecks of bulk-loading massive datasets into Apache Iceberg. The community is sharing battle-tested advice on Spark optimization, the critical importance of sort ordering, controlling file sizes, and the trade-offs between copy-on-write vs. merge-on-read to prevent memory bloat and ensure fast downstream queries.
That’s all for this week! See you in the next edition.