Data EngineeringFeatured

Multi-Source Digital Lending Data Platform

Production multi-source data platform ingesting core banking, sharia-compliant, operational, and portfolio risk data from 5+ partner banks into Aurora PostgreSQL and Amazon Redshift, powering credit risk, portfolio analytics, and Power BI dashboards for a digital lending business

5 source systems, daily full-refresh ingestion

Performance

Zero-downtime atomic table swap for all partner banks

Improvement

Serving 5 partner banks + 3 internal analytics teams

Impact

Tech Stack

Apache AirflowAmazon Aurora PostgreSQLAmazon RedshiftAWS S3AirbytePythonPostgreSQLMongoDBdbt CoreDockerKubernetes (EKS)Power BIParquetFDW (Foreign Data Wrapper)

Problem Statement

A digital lending platform needed a unified data warehouse aggregating core banking data from multiple partner institutions - including conventional banks, sharia-compliant partners, an operational data source, and a portfolio risk system - across heterogeneous PostgreSQL and MongoDB sources into a single warehouse for credit risk models, portfolio analytics, and executive dashboards.

Technical Approach

Built a layered data platform: daily pg_dump/mongodump to S3 Bronze landing zone, Airflow-orchestrated restore into Aurora PostgreSQL staging databases, central federation schema (using Foreign Data Wrappers across all staging DBs), and Amazon Redshift with schema-per-source load strategies. Operational partner data integrated via Airbyte CDC (24h cadence). Data modeling layered as Transient Staging, Core Canonical Data Model, and Semantic Layer with Data Contracts. All Airflow pipelines run on AWS EKS.

Key Results

→4-source ingestion: conventional banking partners (5 banks), sharia-compliant partners, operational data source (Airbyte CDC), portfolio risk source (FDW direct)
→Aurora PostgreSQL cluster as central federation hub - integration schema aggregates all sources via FDW into a single integration point
→Amazon Redshift warehouse with per-source schemas and mixed load strategies (full refresh, incremental delete+reload, Parquet COPY)
→End-to-end data modeling: Transient Staging to Core Canonical Data Model to Semantic Layer plus Data Contracts
→Atomic table swap pattern for partner restores - zero-downtime daily promotion from staging to production schema
→Consumption layer serving Power BI dashboards, credit risk ML models, and portfolio analytics teams
→In progress - Batch Incremental: Replacing full dumps with incremental batch processing, covering both immutable (append-only) and mutable (updated records) data patterns using log-based streaming logic
→In progress - Real-time CDC: Implementing Debezium for Change Data Capture, streaming changes via Redpanda/Kafka, with Apache Flink handling transformations downstream
→Serves 5 Ethiopian partner banks

Pipeline Console

airflow_dag_run (lending_platform_daily_ingestion_dag.py)

>> Terminal ready. Click "Run Pipeline" to simulate orchestration.

Back to all projects