Skip to main content

Installation FAQ

Using AWS Profiles

A named profile is a collection of settings and credentials that you can apply to an AWS CLI command. When you specify a profile to run a command, the settings and credentials are used to run that command. Multiple named profiles can be stored in the config and credentials files.

For detailed information see: Named profiles for the AWS CLI

Using profile:​

After successfully creating Profile (ex. iomete-stage) need to export the profile to the environment before running terraform. ex:

export AWS_PROFILE=iomete-stage

## Then run terraform code as below

terraform init
terraform plan
terraform apply

P.S. Using the profile in terraform code will create additional works (ex. need to declare profile name in all providers).

We recommend exporting the profile to the environment before running the code.

If using only one account (only default profile) no need to export or separate the AWS profile.

Just run the Terraform code.

Public Access Restriction

By default, the EKS API (Kubernetes cluster API) is accessible from anywhere. However, it's important to restrict public access to a selected IP range for security reasons.

The IOMETE control plane connects to and manages the data lakehouse and Spark jobs using the EKS (AWS Kubernetes service) API address. By default, the kubernetes_public_access_cidrs code in the Terraform script is commented out. However, if you need to restrict public access to the EKS API for security or compliance reasons, follow these steps:

note

💡 You will need a static IP address or address range to restrict public access to EKS.

  • Uncomment the kubernetes_public_access_cidrs in the Terraform script.
  • Be sure not to remove IOMETE's IP addresses (18.156.67.183/32 and 54.235.211.34/32). Removing these addresses will prevent the control plane from accessing the cluster, rendering the lakehouse inoperable.
  • Add your static IP address (or address range, such as 5.194.94.20/30) to the your_ip_range/mask section.
note

❗️ Only deployed IP addresses will have access to all resources and EKS instances created by Terraform.

By following these steps, you can restrict public access to your EKS API and ensure that only authorized users can access your IOMETE lakehouse. If you have any questions or need assistance, please don't hesitate to contact our support team.

Define additional administrators

AWS Key Management Service (KMS) is an Amazon Web Services product that allows administrators to create, delete and control keys that encrypt data stored in AWS databases and products.

AWS KMS can be accessed within AWS Identity and Access Management by selecting the "Encryption Keys" section or by using the AWS KMS command-line interface or software development kit

IOMETE customer-stack Terraform module will use or create a KMS key only who run the Terraform code if additional administrator ARN`s not added. (AWS KMS see:https://docs.aws.amazon.com/kms/latest/developerguide/overview.html )

Adding an additional administrator to the system will grant them access to manage Kubernetes resources in EKS. By default, only the creator of the resources has access to Kubernetes. To add additional administrators, include their user Amazon Resource Names (ARNs) when running the Terraform code. It is important to note that when adding additional ARNs, the creators must include their own ARNs in the list to ensure that they retain access to the resources.

tip

Example: additional_administrators = ["arn:aws:iam::1234567890:user/your_arn", "arn:aws:iam::1234567890:user/user2", "arn:aws:iam::1234567890:user/user3"]

Saving terraform state

By default, Terraform writes its state file to your local filesystem. This works well for personal projects, but once you start working with a team, things start to get more challenging. In a team, you need to make sure everyone has an up to date version of the state file and ensure that two people aren’t making concurrent changes.

Remote state solves those challenges. Remote state is simply storing that state file remotely, rather than on your local filesystem. With a single state file stored remotely, teams can ensure they always have the most up to date state file. With remote state, Terraform can also lock the state file while changes are being made. This ensures all changes are captured, even if concurrent changes are being attempted.

Configuring remote state in Terraform has always been an involved process. For example, you can store state in an S3 bucket, but you need to create the bucket, properly configure it, set up permissions, and then ensure everyone has proper credentials to write to it.

As a result, setting up remote state can be a stumbling block as teams adopt Terraform.

How to save in AWS S3

Terraform State File​

To move or save Terraform state files to an Amazon S3 bucket, follow these steps:

  1. Create an S3 bucket in your AWS account in any region.
  2. Add the Terraform code to a configuration file, such as main.tf.

terraform {
  backend "s3" {
    bucket         = "your-bucket-name"
    region         = "aws-region"
    key            = "terraform.tfstate"
  }
}
  1. Run Terraform 
terraform init
terraform plan
terraform apply

If the output that you receive is similar to the example displayed below, then you can be assured that your code has been executed successfully.

info

“Successfully configured the backend "s3"! Terraform will automatically use this backend unless the backend configuration changes.”

terraform init

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Reusing previous version of hashicorp/kubernetes from the dependency lock file
- Reusing previous version of hashicorp/tls from the dependency lock file
- Reusing previous version of hashicorp/aws from the dependency lock file
- Reusing previous version of hashicorp/null from the dependency lock file
- Reusing previous version of hashicorp/helm from the dependency lock file
- Reusing previous version of gavinbunney/kubectl from the dependency lock file
- Reusing previous version of hashicorp/cloudinit from the dependency lock file
- Reusing previous version of hashicorp/local from the dependency lock file
- Reusing previous version of hashicorp/random from the dependency lock file
- Using previously-installed hashicorp/local v2.1.0
- Using previously-installed hashicorp/kubernetes v2.17.0
- Using previously-installed hashicorp/aws v4.53.0
- Using previously-installed hashicorp/helm v2.8.0
- Using previously-installed hashicorp/cloudinit v2.2.0
- Using previously-installed hashicorp/random v3.1.0
- Using previously-installed hashicorp/tls v4.0.4
- Using previously-installed hashicorp/null v3.1.0
- Using previously-installed gavinbunney/kubectl v1.14.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

To verify that the .tfstate file has been saved to your bucket, you can check the contents of the bucket to confirm that the file exists. If the file is present, it means that the state file has been successfully saved.

How to save in Terraform Cloud

Unlike other remote state solutions that require complicated setup, Terraform Cloud offers an easy way to get started with remote state:

  • Step 0 — Sign up for a Terraform Cloud account here
  • Step 1 — An email will be sent to you, follow the link to activate your free Terraform Cloud account.
  • Step 2 — When you log in, you’ll land on a page where you can create your organization or join an existing one if invited by a colleague.

https://www.datocms-assets.com/2885/1560203125-start.png

  • Step 3 — Next, go into User Settings and generate a token.
  • Step 4 — Take this token and create a local ~/.terraformrc file:
credentials "app.terraform.io" {
    token = "mhVn15hHLylFvQ.atlasv1.jAH..."
}

  • Step 5 — Configure Terraform Cloud as your backend

In your Terraform project, add a terraform block to configure your backend:

terraform {
    backend "remote" {
        organization = "my-org" # org name from step 2.
        workspaces {
            name = "my-app" # name for your app's state.
        }
    }
}

  • Step 6— Run terraform init and you’re done.

Your state is now being stored in Terraform Cloud. You can see the state in the UI:

Remote state setup Terraform

Detailed information and learn how to use Terraform cloud see: https://www.hashicorp.com/blog/using-terraform-cloud-remote-state-management