Dataklin — AI-Powered Data Quality & Entity Resolution
Data quality and entity-resolution platform that profiles, validates, deduplicates, and standardizes datasets before handoff — with LLM-generated rules.
Architecture
From raw file upload through profiling, validation, and entity resolution to reviewed golden records.
Async workers
Storage
Auth
Overview
Dataklin is a platform for data engineers to clean up datasets before they're handed off to other teams: profiling, rule-based validation, fuzzy-matched deduplication, and standardization in one pipeline.
Uploaded files land in MinIO and are queued through Redis/RQ for async processing. Each column gets a statistical profile with a 0–100 quality score, then a rule engine validates emails, phone numbers, ID numbers, dates, regex patterns, and cross-column rules — rules can also be generated from natural language via a provider-agnostic LLM integration.
Entity resolution uses blocking keys and fuzzy matching to cluster likely-duplicate records with union-find, with a human-in-the-loop review step to confirm, split, or merge clusters before survivorship rules produce golden records with full audit trails.
Highlights
- LLM-powered natural language → validation rule generation (provider-agnostic)
- Fuzzy matching + union-find clustering for entity resolution
- PII detection and masking (NIK, phone, email, names)
- Drift monitoring with multi-channel alerting (email/Slack/webhook)
- Direct PostgreSQL/MySQL connections plus PDF/CSV quality scorecards
Role
Solo builder — backend, async pipeline, and entity resolution logic
Context
Personal project
Tech stack
More projects
View all →SMILE Platform (UNDP for Kemenkes)
National immunization logistics system in production across all of Indonesia — streaming CDC + batch analytics platform.
LidValid — Unified Data Validation Platform
Full-stack data validation platform with Tiered Validation — cheap aggregate checks across every table, then precise row-level checks only where they fail.
Real-Time Fraud Detection
End-to-end real-time fraud detection pipeline: Flask UI → RabbitMQ → Kafka → Flink → ClickHouse → Grafana.