Welcome back to Data This Week!
This week’s lineup heavily features the convergence of batch, streaming, and agentic workflows, alongside a hard look at the infrastructure we rely on. We explore everything from pushing the limits of clusterless Lakehouse architectures to the metadata overhauls coming in Apache Iceberg v4. We also look at new tooling focused on managing the wild west of AI agents and a community debate on whether the industry is blurring the lines between OLAP and OLTP.
Here are the top reads, tools, and community discussions for senior data folks this week.
📚 Blogs to Read
From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines
Slack recently migrated over 700 production EMR jobs from SSH-based Airflow operators to a REST-based architecture with zero downtime. Direct SSH access to compute clusters creates a massive security surface, causes “zombie” jobs when Kubernetes pods restart, and blocks modernizations like moving to EKS. By leveraging YARN Distributed Shell and a custom gateway named “Quarry,” Slack completely decoupled the client from the execution environment. It’s a highly practical case study for anyone securing and stabilizing legacy Hadoop or Spark orchestrations. Read more →
I Tried to Build an Iceberg Lakehouse Without a Cluster. Here’s What Actually Happened.
Can you run a batch Lakehouse pipeline on AWS without an always-on Spark cluster? Manish Kumar Yadav shares a weekend proof-of-concept using Amazon S3 Tables, DuckDB, and dbt to process data locally and write back to Iceberg. The post covers the real-world friction of this approach—such as writing custom dbt materializations because dbt-duckdb lacks native Iceberg write support, navigating DuckDB’s transaction constraints with CREATE OR REPLACE, and handling AWS signature errors. A must-read if idle compute costs are eating your budget.
Read more →
Apache Iceberg v4: The Current State, the Proposals, and Why They Matter
Alex Merced breaks down the discussions from Iceberg Summit 2026. While Iceberg v4 isn’t finalized, it directly targets the metadata bottlenecks caused by high-frequency streaming commits and massive schemas. The post dives into key proposals like adaptive metadata trees—which enable single-file commits for fast writes while periodically rebalancing into leaf manifests for read performance—and shifting metadata storage from Avro to Parquet. Crucial context for anyone managing Iceberg at petabyte scale. Read more →
Announcing General Availability of Apache Spark 4.0 on Amazon EMR
Spark 4.0 brings massive shifts to EMR workloads, most notably the VARIANT data type, Spark Connect, and expanded ANSI SQL scripting. The VARIANT type (supported natively via Iceberg v3) allows you to ingest semi-structured JSON without defining schemas upfront, applying structure efficiently at query time. Meanwhile, Spark Connect decouples the client and server, allowing developers to debug PySpark natively from local IDEs against serverless EMR compute without the headache of packaging code.
Read more →
Parenting Iceberg and Lance with Gravitino
Mehul Batra shares a hands-on field note exploring the reality behind unified lakehouse architectures. The post details how to put a single Apache Gravitino catalog control plane in front of both Apache Iceberg tables (for BI and tabular analytics) and Lance tables (for multimodal and vector data). It’s a great look at managing centralized metadata and governance across different table formats without creating data silos. Read more →
Introducing Omnigent: A Meta-Harness to Combine, Control and Share Your Agents
Databricks just open-sourced Omnigent to address the fragmentation of the AI agent ecosystem. Instead of juggling Claude Code, Codex, and custom agents in isolated silos, Omnigent acts as a “meta-harness.” It provides a unified API and UI/CLI to compose multiple agents into teams, enforce stateful security and cost policies, and allow real-time collaboration where teammates can securely join a live agent session. Read more →
🛠️ Tools
Bifrost
What it is: An ultra-fast enterprise AI gateway built in Go, benchmarking up to 50x faster than LiteLLM. Bifrost unifies 23+ AI providers behind a single OpenAI-compatible API with less than 100µs of overhead at 5,000 RPS. It comes packed with enterprise features like semantic caching, intelligent load balancing, cost governance, and native Model Context Protocol (MCP) support. Perfect if you need extreme low-latency routing for large-scale agentic workflows.
Omnigent
What it is: Databricks’ new open-source (Apache 2.0) orchestration layer for AI agents. It wraps different agent runtimes into a single interface, supporting cloud sandboxing (via Modal or Daytona) so you don’t have to execute code on your local machine. It allows you to build composite agents using simple YAML and enforce strict guardrails on agent actions.
💬 Community Sentiments
Is the Industry Actually Swinging Back to Postgres?
Over on r/dataengineering, a vibrant discussion sparked around the blurring lines between OLAP and OLTP, triggered by Databricks introducing Lakebase (a managed Postgres engine) and Snowflake acquiring Crunchy Data. The consensus? We aren’t abandoning distributed OLAP for heavy analytics. However, the rise of agentic AI demands unified state and fast transactional capabilities living directly alongside the data lake. For modern apps and AI agents, keeping the OLTP backend natively integrated with the lakehouse avoids complex CDC syncs and fragile data movement pipelines.
That’s all for this week! See you in the next edition.