Event Gateway Service

Overview

The Event Gateway Service introduces a high-performance, Rust-based ingestion API designed for streaming and event-driven data ingestion directly into the IOMETE Lakehouse (Apache Iceberg tables).

It acts as a Firehose-like gateway that provides ultra-low-latency, fault-tolerant data ingestion for logs, metrics, and event streams from distributed systems — bridging real-time data pipelines with the reliability and consistency of IOMETE’s transactional Lakehouse storage.

Planned Features

• Rust-Based High-Performance Gateway: Built for ultra-low latency and high concurrency.
• Native Iceberg Sink: Ingest events directly into Iceberg tables with schema evolution support.
• Multi-Protocol Support: REST, gRPC
• Domain-Isolated Configuration: In line with Data Mesh principles — domains manage their own ingestion endpoints and policies.
• Secure Authentication & Access Control: Token-based and IAM-integrated authentication.

v2 – Scalable, Intelligent Ingestion

• Compression & Encoding Optimization: Adaptive batching and compression to optimize throughput and cost.
• Monitoring & Metrics Dashboard: Real-time visibility into ingestion rates, failures, and latencies.
• Declarative Ingestion Rules: Define transformations or enrichment policies directly at the gateway layer.
• Event Governance & Lineage Integration: Automatic catalog updates, lineage tracking, and observability at event-level granularity.

‍