Metaxy is a pluggable, declarative metadata layer designed to build and maintain complex multimodal data and machine learning pipelines. Developed to solve the common challenges of incremental processing in large-scale ML environments, it enables fine-grained, sample-level versioning. By tracking metadata independently of the heavy raw data (such as images, video, or audio files), Metaxy allows pipelines to evolve while avoiding redundant and expensive recomputations.

At its core, Metaxy manages tabular metadata entries through a unified API that supports various backend storage systems. This architecture ensures developers are not locked into a specific database or storage format, allowing them to switch between development and production environments seamlessly. Whether managing local file-based metadata or scaling to distributed analytical databases, Metaxy provides consistent versioning guarantees and lineage tracking across the computational graph.

Some of the key features are:

• Incremental Processing: Automatically identifies prunable partial data updates to skip unnecessary downstream computations.
• Storage Agnostic: Supports a wide array of backends including Delta Lake, Apache Iceberg, ClickHouse, DuckDB, and BigQuery.
• Pluggable Architecture: Easily integrates with common Python data and AI tooling like Dagster for orchestration, Ray for distributed computing, and SQLAlchemy or SQLModel for database interactions.
• Granular Lineage: Expresses lineage at the field level, allowing independent versioning of fields within the same data sample.
• Consistent Versioning: Uses Merkle Trees to ensure deterministic version hashes across different environments.
• Developer Friendly: Features a clean, typed Python API with syntactic sugar, comprehensive type hints, and Pydantic integration for feature definitions.

Metaxy operates by allowing users to define features as declarative classes. Each feature is mapped to a table in a chosen metadata store, where Metaxy maintains system columns to handle provenance, creation timestamps, and versioning. When a pipeline runs, Metaxy resolves updates by analyzing these version hashes and determining which samples are new, stale, or orphaned. It does not ingest the raw data itself, acting instead as a high-performance index and version control system for your dataset's descriptive attributes.

Some common use cases include:

• Training Data Management: Tracking versions of large video or image datasets used to train generative AI models to ensure experiment reproducibility.
• Incremental Pipeline Updates: Avoiding the re-processing of millions of media files when only a small subset of the metadata or a single feature transformation has changed.
• Cross-Environment Consistency: Using a lightweight storage format like Delta Lake for local iteration and scaling to ClickHouse for production throughput without changing code.
• AI Assistant Integration: Exposing project metadata and feature graphs to LLMs via the Model Context Protocol (MCP) to allow AI agents to understand and interact with the data pipeline state.