The Keeper is a collection of engineering insights, tools, and technical experiments focused on reliable distributed systems, data in motion, and control systems. Created by Vahid Negahdari, a software engineer with a focus on Rust and systems reliability, the project serves as a central repository for projects that solve complex problems encountered in high-scale environments. The platform is designed for engineers and curious minds who need to build robust data pipelines and maintain high system availability under pressure.

The project focuses on delivering high-performance, open-source solutions for data engineering challenges, specifically within the realm of Change Data Capture (CDC) and observability. The Keeper provides tools designed to handle millions of events while maintaining data integrity and system stability. By focusing on the hard problems in distributed systems—such as schema drift, exactly-once delivery, and failover resilience—these projects aim to provide practical, reliable alternatives to standard industry solutions.

Some of the key features are:

• DeltaForge: A high-performance Change Data Capture (CDC) engine written in Rust that streams database changes to Kafka, Redis, NATS, and S3 with built-in safety guards against silent data loss.
• logdrain: An embeddable Rust library and CLI tool that performs online log-template mining, turning millions of noisy log lines into structured, concise templates.
• Schema Sensing: Advanced payload analysis that distinguishes between stable JSON field structures and dynamic keys, allowing for accurate monitoring of schema changes without triggering false evolution alerts.
• Exactly-once per-sink guarantees: A sophisticated delivery architecture that enforces transactional consistency per sink, ensuring reliable data ingestion without sacrificing performance across multi-destination pipelines.
• Resilient Snapshots: CDC snapshot mechanisms that track progress at the table level and verify log reachability, enabling safe handoffs from snapshot to stream with automatic aborts if data integrity is threatened.
• Concurrency Management: Highly optimized Rust implementation that utilizes atomic operations and lock-free paths to ensure the tools scale linearly with CPU core counts while maintaining strict safety guarantees.
• FlowStats: Streaming algorithm utilities providing memory-efficient calculations for cardinality, frequency, quantiles, and sampling within Rust applications.

The tools provided here are intended to be used as high-performance building blocks in modern data infrastructures. Users can integrate them directly into existing pipelines to provide enhanced visibility or to replace legacy systems that struggle with data scale and reliability. The documentation provides deep dives into the trade-offs of each system, explaining how they handle failover, retention, and backpressure.

Some common use cases include:

• Reliable CDC pipelines: Streaming database changes from MySQL or PostgreSQL to multiple heterogeneous downstream sinks while guaranteeing data consistency.
• Log compression: Reducing the volume of noisy log data in large-scale microservices architectures by mining templates and keeping only the essential failure shapes.
• Governance and observability: Automatically detecting the introduction of sensitive fields or new data structures within JSON payloads before they proliferate downstream.
• Incident triage: Using log-template mining to quickly collapse vast amounts of error logs into a ranked list of unique failures for rapid debugging during outages.
• Reducing ingestion costs: Using template-based reduction to lower data egress and storage expenses by only sending schema representatives instead of every raw log line.