Back to Projects
Data EngineeringFeatured
World Bank CDC Analytics Platform
End-to-end analytics engineering pipeline with PostgreSQL OLTP, Debezium CDC, Redpanda, ClickHouse OLAP, dbt marts, Airflow orchestration, and Grafana observability plus a live demo environment.
Live Demo
Real CDC analytics stack running on an ephemeral demo host. GitHub repository is the permanent reference if the demo environment is unavailable.
Open Live Demo Hub~3K rows ingested from paginated REST API
Performance
CDC replication lag monitored with cdc-monitor alerts
Improvement
Live demo environment for portfolio review
Impact
Tech Stack
PostgreSQLDebeziumRedpandaClickHousedbtApache AirflowPrometheusGrafanaDocker
Problem Statement
Demonstrate a vendor-agnostic CDC analytics platform from OLTP ingestion through change capture to governed OLAP marts under constrained resources.
Technical Approach
PostgreSQL source, Debezium to Redpanda, ClickHouse Kafka engine, dbt models, Airflow stage gates, and Prometheus/Grafana CDC drift monitoring.
Key Results
- →Near-real-time CDC from PostgreSQL WAL to ClickHouse with six-stage verification gates
- →dbt staging, analytics mart, and ML feature models with custom data tests
- →Airflow DAGs with failure watchers and idempotent make demo bootstrap
- →Grafana pipeline-health dashboard provisioned from the repository
- →CI/CD with convention gates and remote deploy to live demo host