Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
Make stream processing easier! Easy-to-use streaming application development framework and operation platform.
Build data pipelines, the easy way 🛠️
Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
Implementing best practices for PySpark ETL jobs and applications.
Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
The open document intelligence platform for builders and hackers - DMS for the agentic world
A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton
A comprehensive guide to building a modern data warehouse with SQL Server, including ETL processes, data modeling, and analytics.
A Clojure high performance data processing system
A blazingly fast general purpose blockchain analytics engine specialized in systematic mev detection
Integrate LLM in any pipeline - fit/predict pattern, JSON driven flows, and built in concurency support.
A simplified, lightweight ETL Framework based on Apache Spark
The Supabase of AI era. A modular, open-source backend for building AI-native software — designed for knowledge, not static data.
A Task-Based Workflow Orchestration Engine
An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.
sync/async iterable streams for Python
Flowfile is a visual ETL tool and Python library combining drag-and-drop workflows with Polars dataframes. Build data pipelines visually, define flows programmatically with a Polars-like API, and export to standalone Python code. Perfect for fast, intuitive data processing from development to production.
Service for bulk-loading data to databases with automatic schema management (Redshift, Snowflake, BigQuery, ClickHouse, Postgres, MySQL)
Add a description, image, and links to the etl-pipeline topic page so that developers can more easily learn about it.
To associate your repository with the etl-pipeline topic, visit your repo's landing page and select "manage topics."