The Metadata File is the root of an Apache Iceberg table. Stored in JSON format, this file contains the comprehensive definition of the table. When a query engine asks the Catalog for a table, the Catalog responds with the URI of the table's current metadata file. From there, the query engine has everything it needs to plan and execute the query.
What is in the Metadata File?
A new metadata file is created every time the table state changes - whether data is added, the schema is altered, or a table property is modified. The file contains several critical arrays and identifiers:
- Table Location and Format Version: The base object storage URI for the table and whether the table adheres to Iceberg Spec v1 or v2 (or v3).
- Schema History (schemas): A list of all schemas the table has ever had. The `current-schema-id` points to the active schema. This powers schema evolution, allowing the engine to map old data files to the current schema using stable column IDs.
- Partition Spec History (partition-specs): A list of all physical partitioning layouts the table has used. The `default-spec-id` points to the active spec. This powers partition evolution, allowing old and new partition layouts to coexist.
- Table Properties (properties): Key-value configuration pairs, such as the table's write behavior (Merge-on-Read vs. Copy-on-Write) or snapshot retention policies.
- Snapshot History (snapshots): A list of valid snapshots representing the table's history. Each snapshot entry contains its timestamp, a summary of the operation that created it, and the URI to its Manifest List. The `current-snapshot-id` determines what data the table returns by default.
The Atomic Commit Process
The metadata file is the linchpin of Iceberg's ACID transactions. Because the file is written in JSON and contains the entire table state, a commit operation in Iceberg simply means writing a new JSON metadata file and asking the Catalog to atomically swap its pointer from the old file to the new file.
If two writers try to commit at the same time, they both read the current metadata file, create their own new metadata files, and attempt the atomic swap. The Catalog enforces Optimistic Concurrency Control: the first writer succeeds, while the second writer's swap fails. The second writer must then re-read the newly updated metadata, re-apply its changes, and try again. This ensures that the table state never becomes corrupted or inconsistent.
