Cloud object storage (Amazon S3, Azure Data Lake Storage Gen2, Google Cloud Storage) is the physical foundation of the modern Data Lakehouse. Unlike block storage or network file systems, object storage treats every file as an independent, addressable object with a unique key. It scales to exabytes without capacity planning, costs fractions of a cent per gigabyte per month, and provides eleven nines (99.999999999%) of data durability through automatic replication. These properties make it the only economically viable storage tier for enterprise analytics at scale.
Why Object Storage Is Suited for Analytics
Column-oriented analytics workloads read a small subset of columns from very large datasets. Apache Parquet stores data column-by-column within the file, and object storage supports byte-range GET requests. This combination allows a query engine to read only the specific column data it needs from a large Parquet file without downloading the entire file. For a query that reads three columns from a 100-column, 10 GB Parquet file, only the bytes belonging to those three columns need to transit the network, reducing both latency and data transfer cost dramatically.
The Performance Gap and How Iceberg Closes It
Raw object storage has one significant analytical performance weakness: there is no index. A query engine scanning for all records where region equals "Northeast" must read every file in the table's directory to evaluate the predicate, even if 95% of the files contain no records from that region. Apache Iceberg addresses this with partition-level statistics stored in its manifest files. The query engine reads the manifest before issuing any GET requests, identifies which partitions contain the target region, and issues GET requests only for those partition files. A query that previously required scanning 1 TB of data might scan 50 GB instead.
Storage Classes and Cost Optimization
Not all data is queried equally. Frequently accessed gold-tier tables that power daily dashboards and AI agent sessions should stay in S3 Standard (lowest latency, highest cost per GB). Historical bronze-tier data retained for regulatory compliance or ML retraining might be moved to S3 Intelligent-Tiering or S3 Glacier after a defined retention window, cutting storage costs by 70-90% for infrequently accessed data. Apache Iceberg's table maintenance tooling can automate these tiering moves based on snapshot age, keeping hot data cheap and cold data cheaper.
