Welcome back to Data This Week!
This week’s content is anchored around a central theme: the ongoing modernization of the data lakehouse stack. We have catalog migrations, the stress-testing of DuckDB at serious scale, the strategic end of the Hive Metastore era, and a fascinating framework for rescuing legacy systems using LLMs grounded in domain ontologies. We also revisit why passing dbt tests is not the same as trusting your data.
Here are the top reads, tools, and community discussions for senior data folks this week.
📚 Blogs to Read
CatalogShift: Safely Migrating Apache Iceberg Catalogs Without Moving Data
Switching catalog backends for Apache Iceberg — from Hive Metastore or Glue to a REST catalog — often risks operational downtime or unnecessary data movement. This post walks through strategies to execute catalog-level pointer migrations safely without rewriting underlying Parquet or ORC files. The key insight is that Iceberg separates the catalog pointer from physical data and manifest files, so migrations only require registering existing metadata.json paths in the target catalog. The piece then breaks down step-by-step state verification to ensure snapshot graphs and transaction history remain intact, and closes with a freeze/cutover window pattern using atomic pointer updates to prevent split-brain states across writers during migration.
The DuckDB MySQL Engine at 500 GB: Pushing the Limits
DuckDB’s ability to attach to external relational engines makes it attractive for ad-hoc OLAP queries on OLTP stores. Percona pushes DuckDB’s MySQL extension against a 500 GB workload to rigorously analyse where the boundary sits between direct querying and ETL extract paths. Performance is constrained by MySQL’s storage engine throughput and the single-threaded row-to-column scan conversion happening over local network sockets or disk. Filter pushdowns are critical: query success hinges heavily on how effectively DuckDB translates analytical predicates down to MySQL index scans before pulling data into vectorised batches. The verdict — direct execution works well for targeted aggregation under ~100 GB, but beyond 500 GB, standard CDC pipeline patterns such as Debezium to Iceberg or Parquet remain necessary for high-throughput OLAP.
The Hive Metastore Era is Ending: Close It on Your Terms
HMS is rapidly becoming legacy technical debt in modern data infrastructure. This article outlines a strategic deprecation path toward native Iceberg REST Catalogs such as Polaris, Lakekeeper, and Unity Catalog. The centralized RDBMS-backed HMS model degrades under high concurrent partition-pruning requests, lacks transactional isolation, and enforces rigid schema evolution rules. Moving to the open REST catalog specification decouples compute engines — Spark, Trino, Flink — from catalog implementations, removing metastore lock-in for good. The proposed migration plan is phased: run dual-catalog registration, shift read paths to REST endpoints, then gradually deprecate legacy Hive-serde table definitions.
The Truth About Amazon S3 Tables: My Experiments on Cost, Compaction, and DuckDB
AWS S3 Tables introduces managed Apache Iceberg storage natively within S3 buckets. This hands-on evaluation covers real-world auto-compaction behaviour, pricing models, and query interoperability with DuckDB. The standout finding is that S3 Tables offloads all maintenance — bin-packing small files, manifest rewriting, and snapshot expiration — natively, eliminating custom Airflow or Spark maintenance jobs. Storage cost is slightly higher than standard S3 Standard rates, but is offset by savings on the compute clusters previously dedicated to Iceberg layout maintenance. DuckDB can query S3 Tables via the Iceberg REST API standard, though authentication handling and metadata fetches incur a slight initial latency overhead compared to direct S3 Parquet file scans.
dbt Tests Don’t Say If You Can Trust a Table
Basic assertions like not_null and unique pass easily on garbage data as long as structural rules are not broken. This piece breaks down why standard dbt tests deliver a false sense of data reliability and how to implement genuine semantic data trust. The core argument is that standard assertions check state invariants at run time but miss business logic drifts, missing volume distributions, and cross-table semantic mismatches. Real table trust requires historical trend validation — row count volume variance, distribution skew — rather than binary boolean checks. Critically, data quality logic must act as lineage-aware circuit breakers that halt downstream consumer pipelines before corrupted data reaches consumption layers.
An Ontology + LLM Approach to Legacy Data Modernization
Thoughtworks presents a framework combining domain ontologies with Large Language Models to reverse-engineer legacy database codebases — dense stored procedures, legacy SQL scripts — into modern domain-driven data architectures. Using LLMs alone for code conversion introduces hallucinations and misses business rules; grounding LLM extraction inside an explicit Domain Ontology ensures business logic mapping remains accurate and auditable. Parsing legacy procedural logic into a structured knowledge graph then exposes hidden dependencies, complex transformations, and redundant business rules across systems. The result is a structured workflow to translate legacy batch jobs into modular, maintainable dbt models or streaming pipelines without losing business logic in the process.
🛠️ Tools
Project Nessie: Iceberg Catalog Migrator
What it is: A utility built by the Project Nessie team to execute atomic, batch migrations of Apache Iceberg tables across catalog backends. It migrates table definitions, snapshot histories, and schema evolution logs without copying or altering underlying Parquet files. It supports source and target integrations across AWS Glue, Hive Metastore, JDBC Catalogs, and REST-based spec implementations. Crucially, it includes built-in verification passes to validate snapshot alignment between source and target before cutover, minimising operational risk.
💬 Community Sentiments
Copying Production Data to Dev/Test Environments
Over on r/dataengineering, a thread on a recurring challenge for data teams is generating strong consensus: how do you give developers realistic testing environments without incurring massive storage costs, breaking compliance rules, or leaking PII? The community is converging on three patterns. First, zero-copy branching — teams using Snowflake’s Zero-Copy Clone or Apache Iceberg with Nessie strongly prefer metadata pointer manipulation over physical file copying, giving developers full-scale production snapshots instantly with no storage overhead. Second, PII masking pipelines that read from production and apply deterministic hashing or synthetic obfuscation before materialising a scrubbed test dataset. Third, structural subsetting — slicing a deterministic 5% user sample across all relational foreign key boundaries — which keeps non-prod warehouse costs minimal while still exercising edge cases in query execution.
That’s all for this week! See you in the next edition.