Skip to main content

Internal Spark Catalogs

IOMETE allows users to create new catalogs that function as Iceberg REST Catalogs, managed internally by IOMETE. These catalogs are accessible externally via the path /catalogs/<catalog name> and integrate seamlessly with data security rules enforcing role-based access control (RBAC). A valid session or a PAT token is required for authentication.

Spark Rest Catalogs Solution | IOMETESpark Rest Catalogs Solution | IOMETE

Select Iceberg REST Catalog from the Managed Catalogs section, and the following input fields will appear:

  • Name: The name of the catalog.
  • Warehouse: The URL of an S3-compatible warehouse (IOMETE only supports object stores compatible with Amazon S3 APIs).
Spark Catalog create | IOMETESpark Catalog create | IOMETE

[Optional] Configuring a different S3 Compatible Storage​

By default, IOMETE will use the S3 compatible storage and credentials that are configured by your administrator when installing IOMETE. If you wish to use different credentials or a different storage provider for this catalog, click on the Enable Custom Credentials checkbox. Then, you'll see additional input fields for your credentials.

  • Endpoint - If using AWS S3 you can omit the endpoint, for Minio and Dell ECS storage providers the endpoint is mandatory.
  • Access key
  • Secret key
Spark Catalog create | IOMETESpark Catalog create | IOMETE

[Optional] Configuring Additional Properties and Resource Tags​

You can configure Additional Properties that Catalog Properties that IOMETE will expose for this Catalog. You may also add Resource Tagging.

  1. Additional Properties (Optional)
    • Customize Spark catalog behavior by adding key-value pairs.
    • Click Add property
    • Enter Key and Value
  2. Resource Tags (Optional)
    • Organize IOMETE resources with custom tags.
    • Click Add tag Enter Key and Value
Spark Catalogs | IOMETESpark Catalogs | IOMETE

Validating and Saving​

After filling all inputs click Test Connection to test the setup. This will run a series of tests to validate connectivity and various permissions.

Test Connection | IOMETETest Connection | IOMETE

Click Create or Save to save your settings. The new catalog will appear in catalog list.

Spark Catalogs | IOMETESpark Catalogs | IOMETE
Edit & Delete

Editing and deleting the default catalog is not permitted; you can only make changes to other catalogs.

Using a New Catalog

Make sure to restart relevant Lakehouses and Spark Connect instances to ensure a newly created Catalog can be used

ON THIS PAGE