Data Observability is the ability to understand the state of data within a system at any point in time using metrics, logs, traces, and anomaly detection. Borrowed from the concept of software observability (understanding the internal state of a system from its external outputs), data observability applies the same principle to data pipelines and tables: if you can't see what's happening inside your data, you can't understand or fix it when something goes wrong.
The Five Pillars of Data Observability
Monte Carlo, the company that popularized modern data observability, defines five key pillars:
- Freshness: Is your data arriving on time? If a daily batch pipeline fails silently, the table stops updating, and downstream dashboards show stale data. Freshness monitoring detects this by checking when the last snapshot was committed and alerting if it exceeds the expected interval.
- Volume: Did the expected amount of data arrive? If a Kafka consumer drops 90% of events, row count anomaly detection catches it before analysts build incorrect reports on incomplete data.
- Schema: Did the schema change unexpectedly? Iceberg's schema evolution history provides a perfect audit trail for detecting unauthorized or accidental schema changes.
- Distribution: Are column value distributions within expected ranges? Statistical drift in distributions (e.g., revenue values suddenly including zeros at 10x the normal rate) indicates data quality issues.
- Lineage: Which upstream sources and downstream consumers are affected by a specific table? Understanding lineage allows impact assessment when issues are detected.
Iceberg's Built-in Observability Signals
Apache Iceberg's metadata tables (accessible via SQL as virtual tables like table.snapshots, table.manifests, and table.files) provide rich built-in observability signals. Data engineers can query the snapshot history to monitor write frequency, check column-level statistics for distribution shifts, and identify which operations introduced specific data changes, all without any external tooling.
