Back to Projects
Data EngineeringFeatured
Firebase Analytics Data Pipeline
Automated daily pipeline exporting 600K–800K Firebase Analytics events from Google BigQuery to AWS S3 in Parquet format with Snappy compression
600K–800K events exported daily
Performance
~70% storage reduction via Parquet + Snappy
Improvement
Zero BigQuery storage cost via table expiration lifecycle
Impact
Tech Stack
PythonBigQueryAWS S3ParquetApache AirflowGCPSnappyPandas
Problem Statement
Firebase Analytics events stored in BigQuery accumulate rapidly and incur high storage costs, while downstream AWS-based systems need access to this data in an efficient columnar format.
Technical Approach
Built an Airflow DAG that daily queries BigQuery for Firebase Analytics events, exports them to Parquet files with Snappy compression, uploads to S3, and applies BigQuery table expiration (retention lifecycle) to automatically purge source data and eliminate storage costs.
Key Results
- →Exports 600K–800K events daily with zero manual intervention
- →Cross-cloud data flow from GCP (BigQuery) to AWS (S3)
- →Snappy-compressed Parquet reduces file size by ~70% vs raw JSON
- →BigQuery retention lifecycle eliminates ongoing source storage costs
Pipeline Console
airflow_dag_run (firebase_analytics_export_dag.py)
>> Terminal ready. Click "Run Pipeline" to simulate orchestration.