Spark 4.0 Upgrade

Overview

Adds first-class support for Apache Spark 4 on the IOMETE platform, unlocking faster query execution, improved ANSI SQL compliance, and new cost-saving adaptive execution features.

Planned Features

• Compute Engine Upgrade to Spark 4.0 Core: Spark runtime image upgrade to Spark 4.0 across IOMETE clusters.
• Connectors Upgrade: Compatibility validation for core connectors (Iceberg, JDBC, etc.)
• Migration Toolkit: Ensure seamless migration of user jobs, notebooks, and pipelines from Spark 3.x with minimal code changes.
• Testing & Validation Framework: Performance benchmarking, tuning, and auto-scale enhancements

‍