Welcome to this week’s roundup of the data world. In this edition, we look at Netflix’s massive PostgreSQL migration, novel ways to cut cloud warehouse costs using DuckDB, real-time dashboard architectures, a new multi-engine pipeline framework, and the community’s current stance on the dbt vs. SQLMesh debate.
📚 Blogs to Read
Automating RDS Postgres to Aurora Postgres Migration (Netflix TechBlog)
Netflix details their platform engineering strategy for migrating nearly 400 RDS PostgreSQL clusters to Aurora PostgreSQL. To achieve zero data loss and minimal downtime across hundreds of interconnected services, they built a self-service migration workflow using Aurora Read Replicas. The post goes deep into the weeds of handling inactive logical replication slots, pausing Change Data Capture (CDC) pipelines, and enforcing safe cutover windows—without requiring direct database credentials from users. A masterclass in scaling database infrastructure operations. Read more →
Lower Your Warehouse Costs via DuckDB Transpilation (“Quack Mode”)
If you are feeling the burn of BigQuery or Snowflake compute costs, this is a fascinating read. Max Halford introduces “Quack Mode” via his minimalist orchestrator lea. Instead of entirely migrating off a cloud warehouse, this approach transpiles BigQuery SQL into DuckDB syntax, intelligently pulls necessary dependency tables into a data lake (DuckLake), and runs the heavy compute locally (or on cheaper compute). It’s a pragmatic look at the emerging multi-engine paradigm. Read more →
Real-Time Dashboards with Go and PostgreSQL (Without Polling)
For senior engineers, reaching for a heavy OLAP database or streaming infrastructure for real-time data is often the default, but it introduces massive complexity. This article explores a lightweight alternative: using PostgreSQL’s built-in LISTEN/NOTIFY feature combined with Server-Sent Events (SSE) and an in-memory Go broker. It solves the O(N × M) polling problem, allowing one database event to push to unlimited clients without increasing database load. Read more →
Setting up Airflow Locally on Minikube with Helm
A highly practical, step-by-step guide for deploying Apache Airflow in a local Kubernetes environment using Minikube and Helm. It solves a common developer experience (DX) headache by showing how to mount volumes and Persistent Volume Claims (PVCs) so that locally developed DAGs sync to the Airflow scheduler and webserver in real-time, perfectly mimicking a production K8s deployment. Read more →
🛠️ Tools
RivetSQL
Rivet is a newly emerging declarative pipeline framework built around multi-engine execution. It strictly separates what to compute from how to compute it. You can define your pipeline logic once (using SQL, Python, or YAML) and execute it across DuckDB, Polars, PySpark, or Postgres without changing the code.
Key features include:
- Pre-write Assertions: Catch data quality errors before they hit your target table.
- Interactive REPL: A full-screen terminal UI to explore data and iterate on logic.
- Built-in Offline Testing: Validate transformations using offline fixture data without needing a live database connection.
💭 Community Sentiments
dbt-core vs. SQLMesh in 2026?
With Fivetran now owning both dbt Labs and Tobiko Data (SQLMesh), a lively r/dataengineering thread surfaced the tension between technical superiority and ecosystem pragmatism.
SQLMesh’s Edge: Engineers praise its superior state management, forward-only migrations, and native Python models—features that make it architecturally stronger for complex pipelines.
dbt’s Moat: Despite this, the community still favors dbt-core for its massive ecosystem, analyst-friendly onboarding, and first-class integrations with Dagster and Airflow (via Cosmos)—making it the safer bet for most teams.
The Acquisition Wildcard: With both tools under one roof, most expect dbt to remain Fivetran’s flagship, with SQLMesh’s best ideas gradually absorbed into dbt over time. Read more →
That’s all for this week! See you in the next edition.