E

ETL

Java-based ETL template built with Apache Flink, offering a modular, decoupled structure designed to quickly bootstrap scalable, testable, and reliable pipelines. Out of the box provides an easily adaptable PostgreSQL to MongoDB migration.

Mapping-config driven data migration tool for CSV/JSON/XML/TSV with validation and reporting.

Analyzed decades of historical weather station data (1920–1940) using Hadoop MapReduce. Filtered operable stations, computed descriptive statistics (min, max, mean, median), and produced reports/graphs. Designed modular MRJobs to chain tasks together for scalable processing.

Stack Exchange releases "data dumps" of all its publicly available content roughly every three months via archive.org.

This project is an example and a framework for building ETL for this data with Apache Spark and Java.