Welcome back to Data This Week!
This week’s edition is packed with some genuinely impressive production architecture stories. We have Instacart replacing a split Elasticsearch and FAISS setup with sharded Postgres and pgvector, Netflix rethinking batch compute at scale with Kubernetes-native Kueue, AWS open-sourcing a temporal logic governance layer for AI agents, and a practical guide to distributed DuckDB. We also revisit the growing community frustration with dbt Cloud’s new pricing model.
Here are the top reads, tools, and community discussions for senior data folks this week.
📚 Blogs to Read
How Instacart Built a Modern Search Infrastructure on Postgres
Instacart migrated their massive catalog search away from a split architecture — Elasticsearch for full-text retrieval combined with a separate FAISS service for semantic retrieval — entirely into sharded Postgres instances. The trigger for this change was pain: their highly denormalized Elasticsearch setup was buckling under billions of daily writes driven by price and inventory updates. By normalizing data in Postgres and leveraging pgvector alongside GIN indexes, they eliminated document overfetching. The critical design insight is pushing compute down to the storage layer rather than joining data in the application layer. This shift cut their search latency in half, proving that a well-tuned relational database can handle massive scale when the schema and compute placement are designed carefully.
How Netflix Simplified Batch Compute with Kueue
Netflix transitioned their homegrown Compute Managed Batch (CMB) system to Kueue, a Kubernetes-native job queueing system, to handle millions of batch workloads on their Titus platform. The root problem was that building and maintaining custom queuing logic had become a significant bottleneck. By mapping their internal tenant hierarchy directly to Kueue’s Cohorts and ClusterQueues, they unlocked preemption-based fair sharing. This allows tenants to utilise idle reserved capacity without risking starvation for critical jobs, significantly boosting overall cluster utilisation. The key takeaway is that adopting a well-maintained, Kubernetes-native primitive eliminates an entire class of operational complexity that homegrown solutions inevitably accumulate.
Introducing Dogwood: Runtime Verification for AI Agents
AWS has open-sourced Dogwood, a governance language built on temporal logic that extends Cedar — Amazon’s existing policy language for point-in-time access control. The gap Dogwood fills is significant: while Cedar is excellent for evaluating a single permission decision, enterprise AI agents operating within frameworks like Model Context Protocol (MCP) string together multi-step workflows where the danger lies in the sequence of actions, not any individual action in isolation. Dogwood allows you to enforce rules over ordered sequences of events — requiring a prerequisite action before a sensitive one, enforcing rate limits across a session, or preventing external API calls after an agent has accessed confidential data. If you are building or operating agentic data pipelines, this is a critical read.
Running SQL Concurrently Across Three Remote DuckDB Servers with Quack
A deep dive into distributed query execution using DuckDB. The piece explores how Quack acts as a coordination layer to manage and route concurrent SQL queries across remote DuckDB instances, pushing the boundaries of what local-first OLAP engines can accomplish in a distributed topology. If you have been using DuckDB for single-node analytics and are curious about extending it into a federated or distributed architecture without adopting a heavier query engine, this is a well-structured exploration of the patterns and trade-offs involved.
Using Debezium to Introduce Event-Driven Architecture
A practical look at utilising Debezium for Change Data Capture (CDC). The post breaks down how turning database commits into event streams — typically via Kafka — successfully decouples microservices from each other and from the primary database. This pattern makes it significantly easier to power real-time downstream analytics without placing heavy read queries on production databases. The walkthrough is concrete, covering connector configuration, schema registry integration, and the downstream consumer patterns that make CDC-driven architectures resilient to upstream schema changes.
🛠️ Tools
Dogwood
What it is: The official open-source repository for the AWS-introduced temporal logic policy language described above. Dogwood extends Cedar to enforce rules over sequences of events rather than individual point-in-time decisions. For teams building context-grounding frameworks or orchestrating LLM data agents — particularly those integrating with MCP servers — this tool provides the exact syntax needed to draw safety envelopes around complex, multi-step agentic actions.
Spark Observability Skills
What it is: Recently open-sourced by EmbrasureAI on Reddit’s r/dataengineering, this repository provides a curated collection of debugging, observability, and tuning scripts specifically built for Apache Spark. It covers dependency mismatch detection, lineage tracing, and patterns for refactoring heavy PySpark pipelines. If you regularly work with Spark in production and spend time hunting down performance regressions or mysterious task failures, this is a practical toolkit worth bookmarking.
💬 Community Sentiments
The dbt Pricing Debate: Cloud vs. Core
Over on r/dataengineering, the community is actively debating the financial viability of dbt Cloud’s new pricing model, which meters both seats and the number of models built per month. Frustration is particularly sharp among senior engineers who feel that the UI, CI/CD abstraction, and catalog features of dbt Cloud no longer justify the premium — especially following the recent deprecation of free state-aware orchestration in favour of a paid “dbt state” tier. There is a growing and concrete trend of teams migrating back to dbt Core, orchestrating runs via GitHub Actions or containerised task managers, and using AI coding assistants or open-source alternatives to bridge the catalog and scheduling gaps without the escalating vendor lock-in.
That’s all for this week! See you in the next edition.