A

Apache Spark

An end-to-end ELT and analytics engineering project built with PySpark, PostgreSQL, dbt, and Apache Airflow to process customer quote rating data using the Medallion Architecture (Bronze, Silver, and Gold layers). The pipeline extracts raw data, performs scalable transformations and standardization in PySpark and dbt, orchestrates workflows with Airflow, and produces clean, analytics-ready datasets for reporting, dashboards, and business intelligence

Hybrid Filtering with Spark MLlib

Performance benchmarking tool for ClickHouse Key-Value workloads, featuring a synthetic data generator and a multithreaded runner.

In real time inference simulator using the MVTec dataset, aproach via DINOv2 and Claude Sonnet 3.5 (Amazon Bedrock's API)

End-to-end AWS data lake pipeline for fleet telemetry data using S3, Spark, and Athena. Includes partitioned Parquet ETL, vehicle safety analytics, and SQL queries for overspeed and harsh braking detection.

Distributed geospatial data pipeline using Apache Spark and Apache Sedona to analyze NYC taxi demand hotspots from millions of trip records.

A Project based around a simulated music streaming app (Spotify for example) generating live user events, to practice handling real time streaming of large amounts of data.

The SalesStream dashboards is an application for monitoring and analyzing revenue data in real time. By leveraging the power of Apache Spark and Apache Kafka, this system ensures that financial data is processed efficiently and in a timely manner, providing companies with up-to-date insights into their revenue streams.

POC - Apache Spark

This project aims to predict the outcome of future presidential elections using artificial intelligence predictions.

Implementation of Geo-Temporally Weighted Regression (GTWR) using Apache Spark, Spark ML and Apache Sedona.

Implementation of Geographically Weighted Regression (GWR) using Apache Spark, Spark ML and Apache Sedona.

The "Stage Metrics" plugin for Apache Spark to creating metrics by stage status

sentiment analysis using spark ml library. implemented classic ml models: SVM, Logistic Regression, Naive Bayes and Random Forest. implemented embedding: Word2Vec and TF-IDF. also ensemble and hybrid (ml and lexicon based) methods were implemented

Mirror of https://github.com/zero323/pyspark-asyncactions

Execute Hadoop and Spark applications on the BigData@Polito cluster with a single command