Welcome back to Data This Week!
This week’s lineup covers major structural shifts in how we handle databases and transformation pipelines. From Supabase releasing an “operating system” for Postgres to Netflix’s battle-tested approach to handling wide partitions in Cassandra, the focus is heavily on scale. We also look at the massive architectural shift in dbt Core v2 and the arrival of native graph queries in Postgres 19.
Here are the top reads, tools, and community discussions for senior data folks this week.
📚 Blogs to Read
Multigres v0.1 Alpha: An Operating System for Postgres
Supabase just open-sourced Multigres, an operational layer designed to bring Vitess-grade horizontal scaling and high availability to Postgres. It manages read replicas, failovers, and backup orchestration natively. The standout feature in this v0.1 release is its two-service connection pooling architecture (multigateway and multipooler) that allows transparent traffic routing during failovers. A great read on scaling Postgres beyond single-node limitations. Read more →
Dynamic Repartitioning for Time Series Workloads in Cassandra
Anyone running Cassandra at scale knows the pain of wide partitions causing garbage collection pauses and tail latencies measured in seconds. Netflix engineering shares their architectural fix: an asynchronous pipeline that detects wide partitions on the read path and dynamically splits them into smaller, parallel-readable chunks. It’s a masterclass in distributed systems design, bringing tail latencies down from seconds to low double-digit milliseconds—without downtime. Read more →
dbt Core v2 is Here: Rebuilt on the Rust Fusion Engine
The two-engine era of dbt is over. dbt Core v2 is completely open-source (Apache 2.0) and now shares the same high-performance Rust foundation as the Fusion engine. This update brings massive parse-time improvements for large projects, replaces bulky JSON artifacts with high-performance, easily queryable Parquet files, and streamlines adapter development via ADBC. If you’ve been managing sprawling dbt monorepos, this architectural overhaul is exactly what you’ve been waiting for. Read more →
Handling Graphs with SQL/PGQ in PostgreSQL
PostgreSQL 19 introduces the SQL/PGQ (ISO/IEC 9075-16:2023) standard, enabling native graph queries over regular relational tables without needing extensions or migrating to a dedicated graph database. This tutorial walks through defining a graph as metadata via CREATE PROPERTY GRAPH and traversing multiple hops with GRAPH_TABLE. A huge win for teams looking to run relationship-heavy analytics directly on their transactional stores.
Read more →
Every Data Engineer Needs to Understand IAM
It’s easy to treat AWS IAM as an afterthought or a DevOps problem, but this piece argues it’s a core competency for data engineers. Using an airport checkpoint as an analogy, it breaks down the exact mechanics of access, permissions, and security boundaries. For senior engineers building multi-tenant data platforms or secure data lakes, deeply understanding these boundary conditions is non-negotiable. Read more →
🛠️ Tools
Databow
What it is: A blazing-fast, Rust-built command-line tool for querying databases via ADBC. It provides an interactive SQL shell with syntax highlighting, clean tabular output, and the ability to export results directly to JSON, CSV, or Arrow IPC files. Perfect for local debugging and quick ad-hoc analysis.
ktx (Context Layer for Data Agents)
What it is: An executable context layer designed to make AI agents (like Claude Code or Codex) query your data accurately. Instead of agents hallucinating metrics or rewriting canonical SQL, ktx automatically builds a semantic layer from your warehouse, resolves fan/chasm traps, and serves approved SQL definitions and company knowledge directly to the agent via MCP.
pg_durable
What it is: An open-source extension from Microsoft that integrates durable workflow orchestration directly inside PostgreSQL. Using a custom SQL DSL, it allows you to define complex ETL processes, scheduled jobs, and sequential or parallel tasks (~>, &) that survive crashes and disruptions without needing an external orchestrator.
💬 Community Sentiments
ingestr Rewritten in Go: 12x Faster Ingestion
Over on r/dataengineering, the creators of the ingestr CLI announced a complete rewrite from Python to Go. The community is actively discussing the performance gains—up to 12x faster data ingestion across 100+ sources—by eliminating Python’s overhead and dependency issues, as well as enabling easy cross-compilation for self-contained binaries. The thread also sparked a lively debate on open-source licensing versus source-available models when building competitive data infrastructure startups.
That’s all for this week! See you in the next edition.