Smart Compaction: Predicting Compaction Utility from Lakehouse Table Metadata

In modern data lakehouses, open table formats like Apache Iceberg, Delta Lake, and Hudi provide snapshot isolation by appending immutable files. Over time, this leads to the 'small file problem,' where an accumulation of tiny files degrades query performance. While compaction (rewriting small files into larger ones) is a standard maintenance task, it is computationally expensive. Current systems rely on simple, often arbitrary, thresholds to decide when to compact, lacking a data-driven understanding of when the operation actually provides a net benefit.

The authors developed a simulation framework to generate 2,376 Apache Iceberg tables with varying write patterns, partition skews, and file sizes. They extracted 17 metadata features—such as file counts, partition statistics, and snapshot history—directly from manifest files without reading the actual data. They then trained an XGBoost model to predict the file-reduction ratio (the effectiveness of compaction) and evaluated its performance against a baseline of simple heuristic thresholds.

The study reveals that the binary decision of whether to compact is effectively solved by a simple rule: if a partition contains more than four files, it is a candidate for compaction. For more granular control, the XGBoost regressor predicts the file-reduction ratio with high accuracy (R2 = 0.998). Crucially, this model generalizes well to TPC-H benchmark schemas without retraining. However, the authors caution that compaction is not a universal performance booster; while it accelerates metadata-heavy queries, it can significantly slow down full-scan aggregations by reducing the degree of task parallelism available to the query engine.

This research provides a lightweight, metadata-only approach to optimizing storage maintenance in lakehouses. By demonstrating that complex ML models are not strictly necessary for the binary compaction decision and identifying the specific trade-offs between metadata overhead and task parallelism, the paper offers a practical framework for building more intelligent, cost-aware automated maintenance systems.