Welcome to another issue of Data This Week. In this edition, we’re looking at some major architectural shifts in streaming and lakehouse query routing, Netflix’s approach to graph databases at scale, and what the community actually thinks about Databricks’ new BI migration tool.
Here are the top engineering updates and resources for the week.
📚 Blogs to Read
Introducing Scalable Topics in Apache Pulsar 5.0
If you’ve worked with partitioned-log systems like Kafka or early Pulsar, you know the pain of partition scaling: increasing partition counts breaks modulo hashing, destroying per-key ordering for in-flight traffic. Pulsar 5.0 solves this by introducing Scalable Topics. It shifts from modulo routing to range-based routing using a Segment DAG (Directed Acyclic Graph). This allows topics to dynamically split or merge based on load while preserving strict key ordering across topology changes—without requiring downtime or a complete topic drain. Read more →
Routing Multiple Query Engines with Iceberg
Iceberg standardizes the storage layer, but routing queries to the most cost-efficient or performant engine (Trino vs. DuckDB vs. Snowflake vs. Athena) is usually a manual, fragmented process. This post breaks down the architecture of a SQL routing proxy that dynamically dispatches queries based on protocol, shape, and cost model. It’s an excellent read on how to build a unified control plane that isolates workloads and translates SQL dialects on the fly to prevent compute overspend. Read more →
High-Throughput Graph Abstraction at Netflix: Part I
Netflix handles 10 million operations per second across 650 TB of graph datasets for OLTP use cases—such as real-time topology and social graphs. Instead of building a graph database from scratch, they layered a Property Graph model over their existing Key-Value and TimeSeries abstractions. A key takeaway is their storage strategy: they decouple edge links from edge properties and separate forward/reverse indexes to prevent wide-row partition bottlenecks in Cassandra while maintaining low latency. Read more →
Apache Iceberg Partition Evolution: Change Your Partitioning Strategy Without Rewriting Data
In traditional Hive-style partitioning, switching a 10 TB table from monthly to daily partitions requires a complete, expensive data rewrite. This article provides a highly practical walkthrough of how Iceberg decouples partition specs from file paths to solve this. By treating the partition spec as metadata, new writes adopt the new daily layout while old data remains in the monthly layout. The query planner transparently merges both specs—a “mixed-spec” table—during query execution without any downstream SQL changes. Read more →
Automate Data Discovery and Centralized Management with AWS Glue Data Catalog
Manual governance workflows bottleneck engineering velocity. This post outlines an automated, event-driven architecture using AWS-native services. By leveraging EventBridge to monitor resource creation (such as a new S3 bucket), it triggers Lambda to instantly spin up Glue Crawlers and PII detection jobs. This ensures your Data Catalog is populated with accurate schemas and sensitivity classifications within minutes of a data store being provisioned. Read more →
🛠️ Tools
QueryFlux
What it is: An open-source, Rust-based multi-engine SQL query router highlighted in the LakeOps post above. It sits between your SQL clients and backend engines (Trino, DuckDB, StarRocks, Athena), providing:
- Protocol Translation: Accepts Trino HTTP, Postgres/MySQL wire, Snowflake HTTP, and Arrow Flight SQL.
- Dialect Conversion: Automatically translates queries using Python’s
sqlglot(e.g., seamlessly translating Trino SQL to DuckDB SQL). - Smart Routing & Load Balancing: Routes based on headers, query regex, or client tags with per-cluster concurrency limits and queuing.
- Low Overhead: Written in Rust, it adds roughly ~0.35ms (p50) overhead, keeping it off the critical latency path for analytical queries.
💬 Community Sentiments
Databricks Now Supports Importing Tableau and Power BI Files into Genie Code
Databricks recently announced that Genie Code can now ingest .twb and .pbit files to automatically generate AI/BI Dashboards and map existing measures to Databricks Metric Views.
The Community Takeaway: While the feature is a massive time-saver for scoping migration efforts and translating “vanilla” visualizations, the engineering community is highly skeptical about its ability to handle complex enterprise logic.
- The limitation: A vast majority of enterprise semantic layers rely on complex DAX (in Power BI) or LOD expressions (in Tableau) baked into the BI layer.
- The reality: Genie Code relies on SQL for translation. Because DAX is heavily context-aware and SQL is not, complex metric translation will likely require manual rebuilding. However, users agree it’s a great tool for quickly replacing the 80% of standard dashboards that don’t rely on deep, context-heavy calculations, freeing up engineers to focus on the complex 20%.
That’s all for this week! See you in the next edition.