Automatic Table Maintenance and Compaction

Overview

Managing Iceberg tables today is manual and error-prone. This project introduces automated table maintenance and visibility tools so users can benefit from cleaner tables, storage savings, and faster queries without worrying about low-level operations.

Key Features

• Automated Cleanup: Old snapshots, orphan files, and leftovers are cleaned up automatically with no setup required.
• Central Maintenance Service: Fully managed by the platform, no user pipelines needed.
• Smart Scheduling: Maintenance runs during low-traffic periods that avoid overlapping jobs or conflicts.
• User Flexibility: Users can optionally define maintenance windows & thresholds.
• Self-Healing: Failed maintenance jobs auto-retry in the next safe window.
• Visibility & Impact Tracking:
  • Table optimization status visibility in the catalog, SQL editor, and monitoring dashboards.
  • See storage savings and query performance improvements.
  • Notifications during maintenance (e.g., “Compaction in progress; DMLs may fail for 10 min”).

‍