LidValid — Unified Data Validation Platform
Full-stack data validation platform with Tiered Validation — cheap aggregate checks across every table, then precise row-level checks only where they fail.
Architecture
Tiered Validation: cheap aggregate checks run across all tables first, then precise row-level checks run only where they fail.
Backend
Security
Infrastructure
Overview
LidValid consolidates two of my earlier standalone tools — an aggregate/statistical validator and a row-level validator — into one unified web application with a FastAPI backend and a React SPA frontend.
Its core idea is Tiered Validation: Tier 1 runs five statistical checks (row counts, completeness, uniqueness, column stats, period breakdowns) across every table cheaply; only tables that fail get the expensive Tier 2 row-level comparison, which chunks by primary key to find missing rows and value-level discrepancies.
It supports MySQL, ClickHouse, and PostgreSQL connections with encrypted credentials, per-user RBAC (admin/editor/viewer), real-time progress polling, cancel/resume, and Excel-compatible exports — backed by 147 pytest test cases carried over from the two legacy tools it replaces.
Highlights
- Tiered Validation: aggregate checks first, row-level checks only for failing tables
- Auto-detects primary/sorting keys from DDL, including composite keys
- RBAC with per-user data scoping enforced on every endpoint
- Numeric precision tolerance to avoid false positives from serialization differences
- 147 pytest test cases; encrypted credential storage; Docker + Caddy deployment
Role
Solo builder — backend, frontend, and validation engine
Context
Personal project, unifying two production validation tools
Tech stack
More projects
View all →SMILE Platform (UNDP for Kemenkes)
National immunization logistics system in production across all of Indonesia — streaming CDC + batch analytics platform.
Dataklin — AI-Powered Data Quality & Entity Resolution
Data quality and entity-resolution platform that profiles, validates, deduplicates, and standardizes datasets before handoff — with LLM-generated rules.
Real-Time Fraud Detection
End-to-end real-time fraud detection pipeline: Flask UI → RabbitMQ → Kafka → Flink → ClickHouse → Grafana.