While "UniForm" is often discussed in the context of Databricks' commercial offerings, its foundation lies in the open-source community as the Delta Universal Format. It is a feature native to the open-source Delta Lake project, designed to allow users to expose Delta tables to engines that only speak the language of Apache Iceberg or Apache Hudi.
Open-Source Interoperability
The Delta Universal Format operates on a simple but powerful premise: because Delta, Iceberg, and Hudi all rely on Apache Parquet files for data storage, interoperability is purely an exercise in metadata translation.
When the Universal Format feature is enabled via table properties (e.g., delta.enableIcebergCompatV2 = true) in an open-source Spark deployment, the Delta writer takes on a dual responsibility. Every time a new transaction commits to the Delta log, an asynchronous process writes a corresponding Iceberg metadata tree (v1.metadata.json, manifest lists, etc.) into the exact same table directory.
Bridging Ecosystems
This open-source capability is vital for organizations implementing a multi-engine Lakehouse without locking into a single commercial vendor. For example:
- A data engineering team can use open-source Apache Spark to build complex ETL pipelines that natively write Delta Lake formats, taking advantage of Delta-specific features.
- A downstream analytics team can use an engine like Dremio or Amazon Athena, which query the table using the universally generated Iceberg metadata, oblivious to the fact that the table was originally created as a Delta table.
Architectural Considerations
The Delta Universal Format handles schema evolution, partition mapping, and data typing automatically. However, it is an asynchronous, read-optimized bridge. Because the "source of truth" remains the Delta transaction log, external engines reading the Iceberg metadata will always be highly consistent, but they generally cannot write to the table. Modifications must be routed back through an engine that understands the native Delta format to ensure the dual-metadata generation process remains synchronized.
