D

data-engineering

Fundamental theory and practice in Machine Learning (ML) and Data Science. 🧮

End-to-end design of a Hadoop-based ecosystem for healthcare data at scale (50 TB, IoT streams, medical imaging). Proposed a 10-node cluster architecture integrating HDFS, Spark, Hive, NiFi, Kafka, and Docker with HIPAA-compliant security (Kerberos, TLS, Apache Ranger). Delivered a proof-of-concept Docker deployment and professional proposal document.

A collection of Airflow operators, hooks, and utilities to elevate dbt to a first-class citizen of Airflow.

See the documentation at: https://airflow-dbt-python.readthedocs.io/