Airflow Data Pipeline
Synthetic sales pipeline with Kafka + Airflow + PostgreSQL — full-load batch plus real-time streaming.
Architecture
Synthetic sales data via both full-load batch and real-time Kafka streaming into PostgreSQL, with Airflow orchestrating scheduled transforms.
Orchestration
Ingestion
Infrastructure
Overview
A data pipeline over synthetic sales data combining two ingestion modes: full-load batch and real-time streaming.
Kafka handles streaming ingestion, Airflow orchestrates the batch workflows, and PostgreSQL serves as the destination store.
Highlights
- Both full-load batch and real-time streaming in one pipeline
- Orchestrated with Apache Airflow
Tech stack
More projects
View all →SMILE Platform (UNDP for Kemenkes)
National immunization logistics system in production across all of Indonesia — streaming CDC + batch analytics platform.
LidValid — Unified Data Validation Platform
Full-stack data validation platform with Tiered Validation — cheap aggregate checks across every table, then precise row-level checks only where they fail.
Dataklin — AI-Powered Data Quality & Entity Resolution
Data quality and entity-resolution platform that profiles, validates, deduplicates, and standardizes datasets before handoff — with LLM-generated rules.