Cleanup Untracked Table Folders Job

---

The Cleanup Untracked Table Folders Job helps find table folders that still exist in object storage even though the corresponding tables are no longer active in the catalog.

Use this job when tables were dropped without purging their physical files. It can run in dry-run mode to report candidate folders, and it deletes folders only when deletion is explicitly enabled.

• Version: 0.1.4
• Source: View on GitHub

warning

This job can delete data from object storage. Always run it in dry-run mode first, review the candidates and audit table, and only enable deletion after confirming the output.

Installation

You can deploy this job from the Marketplace or create the job template manually. The Marketplace flow is recommended because it pre-populates the image, main class, and default configuration.

Marketplace

• Open Job Templates from the left sidebar.
• Click Marketplace to open the list of preconfigured Marketplace jobs.
• Find cleanup-untracked-table-folders, open the actions menu, and click Deploy.
• The Marketplace template opens a pre-filled Create New Job form. Review the defaults and update the configuration for your environment.

Manual Setup

Create a job template manually with New Job Template if you do not use the Marketplace flow.

Specify the following parameters when creating the job manually:

Field	Value
Name	cleanup-untracked-table-folders
Docker image	iomete.azurecr.io/iomete/cleanup-untracked-table-folders:0.1.4
Main class	com.iomete.cleanup.untrackedtablefolders.App
Main application file	spark-internal
Config file path	/etc/configs/application.json
Config file name	application.json

Running the Job

The Marketplace template includes a default configuration under the Config Maps tab. Review the databases and safety settings before creating the job.

Dry-Run Example

Use dry-run mode first to inspect candidates without deleting data.

{
  "catalog": "spark_catalog",
  "databases": ["analytics"],
  "exclude_paths": [],
  "exclude_database_folders": [],
  "older_than_hours": 24,
  "dry_run": true,
  "delete_enabled": false,
  "max_candidate_folders_per_database": 10,
  "collect_size_statistics": true
}

Delete-Enabled Example

Enable deletion only after reviewing the dry-run output.

{
  "catalog": "spark_catalog",
  "databases": ["analytics"],
  "exclude_paths": [
    "s3a://bucket/data/analytics/protected_table"
  ],
  "exclude_database_folders": [],
  "older_than_hours": 24,
  "dry_run": false,
  "delete_enabled": true,
  "max_candidate_folders_per_database": 5,
  "collect_size_statistics": true
}

Configuration Fields

Field	Default	Description
catalog	spark_catalog	Catalog to inspect.
databases	required	List of databases to scan. The job does not scan all databases automatically.
exclude_paths	[]	Full object-storage paths that must never be selected as candidates.
exclude_database_folders	[]	Database-relative folder exclusions in the format database.folder.
older_than_hours	24	Candidate folders must be older than this cutoff.
dry_run	true	Reports candidates without deleting data.
delete_enabled	false	Must be true together with dry_run=false before deletion can happen.
max_candidate_folders_per_database	10	Safety limit. If more candidates are detected for a database, the database is skipped.
collect_size_statistics	true	Estimates candidate and deleted-folder sizes for audit reporting.

Recommended Workflow

1. Run with dry_run=true and delete_enabled=false.
2. Review the driver logs and audit table output.
3. Add exclude_paths or exclude_database_folders for anything that must be protected.
4. Keep max_candidate_folders_per_database low for the first delete-enabled run.
5. Enable deletion only after the dry-run output is reviewed.
6. Query the audit table after the run.

Safety Controls

The job is designed to fail closed. If it cannot prove that a path is safe to delete, it does not delete it.

Safety check	Purpose
dry_run=true by default	Reports candidates without deleting data.
delete_enabled=false by default	Requires explicit confirmation before destructive cleanup.
Explicit database list	Limits cleanup scope to configured databases.
Scan-root validation	Prevents scanning outside the database storage boundary.
Active table protection	Protects folders that are active table locations or contain active table locations.
Pre-delete recheck	Re-discovers active table locations immediately before deletion.
exclude_paths	Protects explicitly configured storage paths.
exclude_database_folders	Protects database-relative folders without requiring full object-storage paths.
Sentinel folder exclusion	Skips Spark/Hadoop staging folders such as _temporary and .spark-staging*.
older_than_hours	Filters out folders newer than the configured cutoff.
max_candidate_folders_per_database	Skips cleanup when too many candidates are detected.
Empty-database guard	Skips databases with no active table locations by default.
Catalog-scoped storage access	Uses the selected catalog's own storage settings, and fails closed if they cannot be resolved.
Unresolved ownership guard	Blocks deletion for a database when any table's storage location cannot be read.
Audit table	Records success, skipped, and failed outcomes for each database.

Object Storage Access

You do not configure storage settings or credentials for this job. Cleanup always uses the storage configuration associated with the catalog you selected in catalog.

If that storage configuration cannot be resolved safely, the job stops for that database and records FAILED. It does not fall back to any other storage configuration. This helps ensure cleanup remains within the storage associated with the selected catalog.

Excluding Paths

Use exclude_paths when you know the full object-storage path to protect:

{
  "exclude_paths": [
    "s3a://bucket/data/analytics/manual_archive"
  ]
}

Use exclude_database_folders when you want to protect a folder relative to a configured database:

{
  "exclude_database_folders": [
    "analytics.manual_archive"
  ]
}

Both forms are included in the effective excluded-path list written to the audit table.

Audit Table

The job writes audit rows to:

spark_catalog.iomete_system_db.cleanup_untracked_table_folder_runs

Use the audit table to review candidates, skipped databases, deleted folders, runtime metadata, and failure details.

Important Audit Columns

Column	Meaning
run_id	ID shared by all database rows from the same job execution.
spark_app_id	Spark application ID for the job run.
runtime_compute_id	Platform runtime compute/run identifier, when available.
runtime_compute_namespace	Kubernetes namespace where the Spark driver ran, when available.
runtime_domain	IOMETE domain from the runtime environment, when available.
runtime_user	Spark runtime/run-as user from the driver environment, usually SPARK_USER.
external_job_id	Stable platform Job ID shown in the UI, when exposed by the platform runtime.
platform_started_by	Platform user who started this specific job run, when exposed by the platform runtime.
catalog_name	Catalog scanned by the job.
database_name	Database scanned by this audit row.
operation	Cleanup operation name.
dry_run	Whether the run was report-only.
delete_enabled	Whether destructive deletion was explicitly enabled.
older_than_hours	Age threshold used for candidate detection.
cutoff_time	Timestamp cutoff derived from older_than_hours.
max_candidate_folders_per_database	Candidate-count safety limit used for this run.
excluded_paths	Effective excluded paths used during candidate detection.
status	SUCCESS, SKIPPED, BLOCKED, or FAILED.
status_reason	Machine-readable reason for skipped or failed outcomes.
error_message	Error details for failed or intentionally skipped outcomes.
discovered_database_location	Database location discovered from the catalog.
storage_scan_location	Object-storage root scanned by the job.
active_table_count	Number of active tables discovered for the database.
unresolved_table_count	Number of tables whose storage location could not be read during discovery.
storage_folder_count	Number of immediate storage folders discovered.
candidate_folder_count	Number of folders confirmed as untracked and eligible for deletion.
potentially_untracked_folder_count	Number of folders with no verified catalog owner, reported for review only when table storage ownership could not be fully verified.
deleted_folder_count	Number of folders deleted.
candidate_folders	Candidate folder paths.
deleted_folders	Deleted folder paths.
candidate_object_count	Estimated object count across candidate folders when size statistics are collected.
candidate_total_size_bytes	Estimated candidate size when size statistics are collected.
deleted_object_count	Object count for deleted folders when size statistics are collected.
deleted_total_size_bytes	Deleted size when size statistics are collected.
diagnostic_details	Additional contextual metadata for troubleshooting, such as path samples and truncation flags.
start_time	Database processing start time.
end_time	Database processing end time.

Runtime Identity Fields

The audit table separates who the Spark job runs as, who started the run from the platform, and which platform job/run the audit row belongs to.

Field	Example	Source	Meaning
runtime_user	hasan	SPARK_USER from the driver environment	Spark runtime/run-as user. This is the user the Spark job executes as.
platform_started_by	fde_admin	IOMETE_JOB_STARTED_BY, when exposed by the platform runtime	Platform user who started this specific run in the UI or platform backend. This can differ from runtime_user.
external_job_id	3e9342e7-89d3-42f8-a603-e00b783cca17	IOMETE_EXTERNAL_JOB_ID, when exposed by the platform runtime	Stable platform Job ID for the job definition/template. This stays the same across multiple runs of the same job.
runtime_compute_id	afceb977-cd1e-4ff8-89e3-2f7876436ecf	IOMETE_COMPUTE_ID, when available	Specific platform runtime/application identifier for one execution. This changes for each run.
runtime_compute_namespace	spark-resources-1	IOMETE_COMPUTE_NAMESPACE, when available	Kubernetes namespace where the Spark driver ran.
runtime_domain	fde	IOMETE_DOMAIN, when available	IOMETE domain associated with the runtime.
spark_app_id	spark-ffbd89771a0a4a248b947676b943557e	Spark application context	Spark application ID used to correlate with Spark logs/history.
run_id	2edb19ec-8cb0-4d27-8965-ffb4912bfa60	Generated by this cleanup job	Cleanup job run ID. It groups all per-database audit rows from the same execution.

Example audit query:

SELECT
  start_time,
  database_name,
  status,
  status_reason,
  dry_run,
  delete_enabled,
  candidate_folder_count,
  deleted_folder_count,
  runtime_user,
  platform_started_by,
  external_job_id
FROM spark_catalog.iomete_system_db.cleanup_untracked_table_folder_runs
ORDER BY start_time DESC
LIMIT 20;

Status Values

Status	Description
SUCCESS	The database was processed normally. Candidates may or may not have been found.
SKIPPED	The job intentionally refused to process or delete for a safety reason.
BLOCKED	Reconciliation completed far enough to inspect catalog and storage state, but a safety condition prevented cleanup. The findings are still reported.
FAILED	An unexpected failure occurred while processing the database.

Common status_reason values include:

Reason	Description
database_not_found	Configured database does not exist or cannot be discovered.
database_location_missing	Database exists but has no usable storage location.
no_active_tables_in_database	Database has no active table locations, so cleanup is skipped by default.
too_many_candidate_folders	Candidate count exceeded max_candidate_folders_per_database.
unresolved_catalog_ownership	At least one table's storage location could not be read, so no folder in the database could be confirmed as untracked.

Unresolved Table Storage Ownership

A table is unresolved when it exists in the catalog but the job cannot read its Iceberg metadata or storage location during the run. This does not mean the table is unhealthy or unwanted. It only means the job does not know which storage folder that table owns.

Storage reconciliation still runs and reports what it found. What changes is what the job is allowed to conclude. If any table in the database is unresolved, that table could own any of the unmatched folders, so no folder in that database can be proven untracked. For that database:

• Every unmatched folder is reported as potentially untracked, not as a candidate.
• potentially_untracked_folder_count carries those folders, and candidate_folder_count stays 0.
• Nothing is deleted, in dry-run mode or in deletion mode.
• The audit row is written with status=BLOCKED and status_reason=unresolved_catalog_ownership.

Other configured databases are unaffected and continue to run.

Understanding Summary Logs

At the end of each database, the job prints a CLEANUP REPORT block in the driver logs. Its sections are:

Report section	Source	Meaning
CATALOG	Catalog metadata	Counts of tables found, tables with verified storage locations, and tables whose storage location could not be read.
TABLES WHOSE STORAGE LOCATION COULD NOT BE RESOLVED	Catalog metadata	Tables that exist in the catalog but whose Iceberg metadata or storage location could not be read. Shown only when there are any.
STORAGE	Object-storage scan	Counts of folders discovered under the scan root, folders belonging to verified catalog tables, and folders with no verified owner.
STORAGE FOLDERS PROTECTED BY A VERIFIED CATALOG TABLE	Catalog metadata	Active table locations discovered from the catalog. These are the protection set and must not be deleted. This list may include paths outside the selected storage scan root.
STORAGE FOLDERS EXCLUDED BY CONFIGURATION	Configuration	Paths protected by exclude_paths or exclude_database_folders.
UNTRACKED STORAGE FOLDERS	Candidate detection	Folders that were not protected and passed the cleanup filters. These are the folders the job would delete in deletion mode.
POTENTIALLY UNTRACKED STORAGE FOLDERS	Candidate detection	Shown instead of the section above when a table's storage location could not be read. These folders have no verified catalog owner, but they cannot be proven untracked, so none of them are deletion candidates.
DELETION	Deletion result	Folders actually deleted when dry_run=false and delete_enabled=true, or BLOCKED with the reason when deletion was prevented.

In short, catalog active table locations come from catalog metadata, while storage folders come from the physical object-storage scan.

Summary

Use this job to review and optionally clean table folders that remain in object storage after tables are no longer tracked by the catalog. Always start with dry-run mode, review the audit table, and enable deletion only after confirming that the candidates are safe to remove.