Welcome back to Data This Week!
This week’s lineup covers major structural shifts in how we handle semantic layers, govern multi-format data lakes, and scale AI infrastructure. From the Apache Software Foundation incubating a new open standard for business semantics to HubSpot’s battle-tested approach for managing 20 billion vectors in Qdrant, the focus is heavily on interoperability and massive scale. We also look at cloud-native financial search architectures and practical ways to finally version control Power BI with Git.
Here are the top reads, tools, and community discussions for senior data folks this week.
📚 Blogs to Read
Running Hermes at Razorpay: A Network-Isolated AI Second Brain
Razorpay engineering details the architecture behind Hermes, their internal AI “second brain.” To ensure strict security across the organization, they deployed a zero-trust architecture featuring a custom auth router, one dedicated Kubernetes pod per employee for absolute tenant isolation, and the Linux kernel (via eBPF) acting as an egress firewall to control external access. A great breakdown on scaling AI applications while strictly enforcing tenant boundaries. Read more →
Cost-Optimized Cloud-Native Financial Search Architecture
A deep dive into replacing memory-heavy search clusters (like Elasticsearch) with a cloud-native architecture over object storage. By integrating Apache Iceberg for storage, Spark Structured Streaming for writes, Trino for read-hydration, and Turbopuffer for operational vector indexing, the author demonstrates a highly efficient two-stage Top-K retrieval pattern. This setup drastically reduces infrastructure costs by leveraging object storage instead of expensive memory-resident instances. Read more →
Lakekeeper Generic Table API Design
The Fresha data engineering team walks through the design of the Generic Table API in Lakekeeper, allowing Iceberg REST catalogs to manage non-Iceberg formats (like Lance). Rather than re-implementing format specifics, the API acts strictly as a governance boundary—handling identity, namespaces, RBAC, and credential vending—while leaving the bytes to native engines. A crucial read for platform teams managing multi-format lakehouses looking to centralize access control without data lock-in. Read more →
Running dbt on Amazon EMR Serverless with Spark Connect
This post explores the modern decoupled approach to running dbt pipelines by combining Amazon EMR Serverless, Spark Connect, and S3 Tables. Spark Connect allows the dbt execution environment to run remotely without heavy client-side dependencies, while EMR Serverless handles automatic compute scaling. Coupled with the performance of Iceberg-backed S3 Tables, this architecture simplifies infrastructure management and optimizes query performance for sprawling data transformations. Read more →
Meet Apache OSSIE: The Open Semantic Interchange
Apache OSSIE has officially entered the ASF incubator, aiming to standardize the semantic layer the way Iceberg standardized table formats. Instead of business logic and metrics being locked in proprietary BI tools, OSSIE provides a vendor-neutral, machine-readable specification to define metrics, entities, and dimensions. With the rise of AI agents querying data, establishing a unified, tool-agnostic source of truth for business definitions is more critical than ever. Read more →
Building the AI Retrieval Infrastructure Behind 20 Billion Vectors
HubSpot engineering outlines how they scaled their Vector as a Service (VaaS) platform to support over 20 billion vectors using Qdrant. Transitioning from a manual Helm-based setup, they built a dedicated Kubernetes Operator framework to automate cluster lifecycle management, safe shard eviction, and replication factor recovery. The post highlights the necessity of custom sharding and dynamic shard balancing to evenly distribute memory loads and control costs at massive scale. Read more →
🛠️ Tools
OpenLineage
What it is: An open-source industry standard framework for data lineage collection and analysis. It provides a common API to capture lineage metadata from across the data ecosystem (Spark, dbt, Airflow) as pipelines run, enabling teams to trace data origins, troubleshoot failures, and maintain strict governance compliance without building custom parsers.
Qdrant
What it is: A high-performance, open-source vector search engine built in Rust. It utilizes the HNSW algorithm for fast approximate nearest neighbor search and offers advanced features like named vectors, payload filtering, and on-disk storage options. It’s highly tunable for balancing precision and cost-efficiency, making it a go-to for massive-scale semantic search and RAG architectures.
💬 Community Sentiments
Using Git for Power BI Reports
Over on r/dataengineering, the community is actively discussing best practices for version-controlling Power BI reports using Git. The consensus strongly points to abandoning binary .pbix files in favor of the newer developer-friendly .pbip (Power BI Project) format, which stores the report and semantic model as text. Teams highlight the importance of disabling auto date/time, adding cache.abf to the .gitignore, and modularizing reports into smaller assets to minimize painful merge conflicts.
That’s all for this week! See you in the next edition.