Data Lineage

Overview

The Data Lineage initiative provides end-to-end visibility into data flow across the IOMETE Lakehouse — from ingestion and transformation to consumption.

It allows teams to understand the full lifecycle of data assets, trace dependencies across domains, and ensure trust, compliance, and operational transparency.

Built natively into IOMETE’s metadata and governance layers, lineage connects SQL queries, pipelines, datasets, dashboards, and event gateways into a unified, queryable graph — empowering both technical and business users to understand where data comes from, how it’s used, and what changes affect it.

Planned Features

• Automatic Lineage Extraction: Capture lineage metadata from SQL queries, ingestion jobs, and transformation pipelines.
• Visual Lineage Graph: Interactive graph view in the Workspace showing upstream and downstream dependencies for tables, views, and dashboards.
• Domain-Aware Lineage: Lineage scoped and visualized within Data Mesh domains to maintain ownership and isolation.
• Change Impact Analysis (Basic): Identify downstream objects affected by schema or table changes.
• Integration with Metadata Catalog: Link lineage data to datasets, owners, and governance policies.
• Cross-Domain Lineage Visualization: End-to-end tracing across multiple data domains with access-controlled visibility.
• Pipeline & Job Lineage: Trace transformations from Event Gateway ingestion through Spark jobs to BI dashboards.
• Time-Travel Lineage: Explore lineage snapshots at any point in time for audits or debugging.
• Lineage API & Query Interface: Expose lineage graph through APIs for integration with external tools.
• AI-Powered Lineage Insights: Automatically generate narratives like “This KPI dashboard depends on datasets X, Y, and Z — last updated 3h ago.”
• Proactive Data Quality Alerts: Notify users when upstream data changes could affect trusted datasets or dashboards.
• Cross-Platform Integration: Ingest lineage data from external systems (dbt, Airflow, Kafka, etc.) for full enterprise visibility.
• Collaboration & Sharing: Allow teams to comment, document, and tag lineage paths within the UI.

‍