Automated data testing applies software testing principles to data: just as application developers run unit and integration tests on every code change, data engineers run data quality tests on every pipeline execution. Catching data quality regressions at the pipeline level prevents bad data from propagating to downstream dashboards, reports, and ML models where the impact is discovered much later and much more expensively.
Testing Frameworks for Iceberg
The primary automated data testing frameworks for Iceberg lakehouses are: dbt tests (not_null, unique, accepted_values, relationships) run as SQL queries against Iceberg tables after each dbt model materialization. Great Expectations defines expectation suites as YAML configuration files, runs them as Spark or SQL jobs against Iceberg tables, and generates HTML data docs showing pass/fail results and data statistics. Soda Core provides a declarative YAML syntax for data quality checks that integrates with Airflow and dbt. All three tools support integration with CI/CD pipelines: a failing data test in CI blocks the pipeline from promoting bad data to production Iceberg tables.
