E

ETL

Configuration and data workflows for an instance of Apache Airflow for the DDRplatform

DataRider bloc for ETL Stream with Spark+Scala

End-to-end AWS data lake pipeline for fleet telemetry data using S3, Spark, and Athena. Includes partitioned Parquet ETL, vehicle safety analytics, and SQL queries for overspeed and harsh braking detection.

Plain text boilerplate removal using character n-gram frequency across a corpus. Builds a template model from a sample, filters files in a single linear pass, and validates automatically. Includes an obfuscated mode where the model is a set of integers and output filenames are hashed: the operator never reads the content. AWK for character processing, Bash for orchestration, Lisp layer planned for positional classification.

Solución end-to-end para la migración y análisis de datos utilizando Python, FastAPI, Kafka y PostgreSQL. Implementa un pipeline de datos asíncrono y una API RESTful para analíticas, todo completamente containerizado con Docker Compose para un despliegue fácil y reproducible.

Unified project demonstrating both batch analytics and real-time streaming pipelines with Apache Spark:

Batch (PySpark/Jupyter): Processed S&P 500 stock data, applied transformations, and ran distributed computations.

Streaming (Spark + Kafka): Built a streaming pipeline to consume Kafka topics, process messages in real-time, and visualize outputs.

Deployed using Docker and Jupyter for reproducibility.

Simulation of a real hospital scenario with a ML model in production

Crawl and extract Home Depot's schema.org/Products.

Official weather data ETL for wind energy project evaluation in El Calafate, Argentina.

Live data source that can be used for data engineering, data warehouse and etl development.

Live data source that can be used for data engineering, data warehouse and etl development.

Live data source that can be used for data engineering, data warehouse and etl development.

A Python extract, transform, load (ETL) pipeline for the CityPulse Smart City dataset.

Amaxa is a new data loader and ETL (extract-transform-load) tool for Salesforce, designed to support the extraction and loading of complex networks of records in a single operation.

Primary repo now on GitHub: https://github.com/davidmreed/amaxa

Stack Exchange releases "data dumps" of all its publicly available content roughly every three months via archive.org.

This project is an example and a framework for building ETL for this data with Apache Spark and Java.

BusinessMatrix ETL 调度平台