Skip to main content

IOMETE Community Edition Deployment on AWS

You're about to install the IOMETE Lakehouse Platform Community Edition on AWS. The IOMETE Community Edition is the most generous Data Lakehouse Platform on the market. Enjoy the benefits of a data lakehouse platform with no restrictions on data volume, users, or queries.

IOMETE Community Edition is supported by the community. Feel free to join the IOMETE Community Discord Server for support and discussions.

Installing the IOMETE platform on AWS should take approximately 25 minutes
  • 15 minutes to create the infrastructure (EKS cluster, VPC, etc.) using Terraform
  • 5 minutes to install the IOMETE platform using Helm

Installation Steps​

Steps

  1. Terraform - Create the necessary infrastructure on AWS for IOMETE to run.
  2. Deploy IOMETE Data Plane - Deploy the IOMETE Data Plane using Helm.
  3. Configure Ingress Gateway - Configure the ISTIO Ingress Gateway to access the IOMETE Data Plane UI.

Prerequisites

1. Terraform​

Reference: https://registry.terraform.io/modules/iomete/iomete-data-plane/aws/latest

Terraform is used to create the necessary infrastructure on AWS for IOMETE to run. It will create AWS resources like VPC, EKS cluster, Roles and Policies, etc.

Prepare Terraform Script​

# 1. Create a directory (e.g. iomete-aws-deployment)
mkdir iomete-aws-deployment

# 2. Change to the directory
cd iomete-aws-deployment

# 3. Download the Terraform script setup
wget https://raw.githubusercontent.com/iomete/terraform-aws-iomete-data-plane/main/examples/deployment-example/main.tf

The main.tf file you downloaded looks like this:

main.tf
loading...

Update the main.tf file with the necessary variables. Required variables are region, cluster_name, and lakehouse_bucket_name. You can find the all available input variables in the module documentation.

Please also check the following specific settings

Apply Terraform​

Run following commands to apply the Terraform script, and create the necessary infrastructure on AWS:

terraform init --upgrade
terraform apply

Please make sure that you have the necessary permissions to create the resources in your AWS account. Please see for Required Permissions to Deploy IOMETE on AWS for more details.

Once terraform apply is complete, you will see the output similar to the following:

eks_update_kubeconfig_command = "aws eks update-kubeconfig --region us-east-1 --name test-deployment1"

Copy and run the command to connect to the EKS cluster using kubectl:

aws eks update-kubeconfig --region us-east-1 --name test-deployment1

At this point, your AWS infrastructure is ready for IOMETE Data Plane deployment. Continue to the next step to deploy IOMETE Data Plane.

2. Deploy IOMETE Data Plane​

Prepare Database​

IOMETE requires a PostgreSQL database to store metadata and other information. Refer Backend Databases for more details.

You can use your own database, or you can use the provided postgresql database.

info

This postgresql database is for testing purpose only. For production, please use your own database that is optimized for production use.

Add bitnami helm repo if you haven't done so.

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

Deploy postgresql database, and wait for it to be ready.

helm upgrade --install -n iomete-system \
  postgresql bitnami/postgresql \
  -f https://raw.githubusercontent.com/iomete/iomete-deployment/main/database/postgresql/postgresql-values.yaml

Wait for postgresql pod to be ready. It takes about ~1 minute

kubectl get pods -n iomete-system \
  -l app.kubernetes.io/name=postgresql --watch

Deploy IOMETE Data Plane Helm Chart​

Add, IOMETE helm repo:

helm repo add iomete https://chartmuseum.iomete.com
helm repo update

See the IOMETE Data Plane Helm - AWS Community Version for more details about available configurations. See here for the data-plane-values.yaml

tip

You don't need to customize values for a default installation. However, if you want to tailor the installation to your needs (perhaps you're using your own database and distinct credentials), then you can override the default values.

Deploy IOMETE Data Plane:

helm upgrade --install -n iomete-system iomete-data-plane \
  iomete/iomete-data-plane-community-aws --version 1.11.2

Wait for IOMETE Data Plane pods to be ready. It takes about ~6 minutes to get everything ready in the first time installation.

kubectl get pods -n iomete-system --watch

3. Configure ISTIO Ingress Gateway​

Please follow the Configure ISTIO Ingress Gateway to configure the Ingress Gateway for IOMETE Data Plane to be able to access the UI.

4. Access IOMETE Data Plane​

Once, IOMETE Data Plane is deployed and ingress gateway is configured, you can access the IOMETE Data Plane UI.

Get the external IP of the ISTIO Ingress Gateway:

kubectl get svc istio-ingress -n istio-system

From the output, copy the EXTERNAL-IP value, and open it in your browser http://EXTERNAL-IP

info

For the first time use username and password from the adminUser configuration (See data-plane-values.yaml). Default values are admin and admin. On your first login, you will be asked to change the temporary password.

That's it! You've successfully set up IOMETE data plane. If you encounter any issues or have any questions please join the IOMETE Community Discord Server for support and discussions.

Quickstart Resources​

Start using IOMETE with the following guides

Sync data from JDBC Databases

Read our guide on how to sync data from JDBC sources, like MySQL, PostgreSQL, and Oracle.

Querying Files in AWS S3

If you have data files in AWS S3, you can directly query them using the S3 connector.

Getting Started with Spark Jobs

Learn how to run Spark jobs on IOMETE.

Getting started with DBT

Learn how to use DBT with IOMETE.

ON THIS PAGE