Apache Comet

Overview

Comet is a Spark plugin that swaps Spark's row-based physical operators for Arrow-native ones backed by the DataFusion (Rust) engine, with a JVM fallback for anything unsupported. It needs no query changes - you turn it on via config. It delivers roughly a 2x speedup on TPC-DS at SF 1000, i.e. about 50% cost savings, but that's a scan/join/aggregation-heavy benchmark - your mileage depends entirely on how much of your plan stays native vs. falls back to Spark.

‍