Skip to main content

Spark Catalogs

IOMETE offers a feature for configuring Spark Catalogs interface.

Catalog list

To view Spark Catalogs, navigate to the Settings menu item and switch to the Spark Catalogs tab. In the list below, you'll find the Spark catalog. The default catalog created by the system is called spark_catalog.

Spark Catalogs | IOMETESpark Catalogs | IOMETE

Add catalog

To add new catalog item, click Add catalog button. You'll see following inputs:

  • Name
  • Lakehouse directory
  • Type - Currently, only 'iceberg' type is supported.
Spark Catalog create | IOMETESpark Catalog create | IOMETE

For custom credentials, click on the Configure credentials radio button. Then, you'll see additional input fields for your credentials.

  • Endpoint - If using AWS S3 you can omit the endpoint, for Minio and Dell ECS storage providers the endpoint is mandatory.
  • Access key
  • Secret key
Spark Catalog create | IOMETESpark Catalog create | IOMETE

After filling all inputs click Add.

Spark Catalogs | IOMETESpark Catalogs | IOMETE
Edit & Delete

Editing and deleting the default catalog is not permitted; you can only make changes to other catalogs.

Catalog in SQL Editor.

Go to the SQL Editor to view the Spark catalog that has been created. Check out the Database Explorer section in the left panel. If you don't see the created catalog, please refresh by using the button. In the given image, we have already created some databases and tables inside the catalog.

Spark Catalogs in SQL Editor | IOMETESpark Catalogs in SQL Editor | IOMETE