P

pyspark

Tumult Core is a collection of composable components for implementing algorithms to perform differentially private computations.

Built a complete machine learning pipeline in SparkML using the Adult Census dataset (~48k rows, 14 features). Implemented data preprocessing, feature encoding, cross-validation, and model training with Logistic Regression and Random Forest. Evaluated models with metrics such as AUC and F1-score. Reflected on scalability trade-offs and optimizations in distributed ML.

Unified project demonstrating both batch analytics and real-time streaming pipelines with Apache Spark:

Batch (PySpark/Jupyter): Processed S&P 500 stock data, applied transformations, and ran distributed computations.

Streaming (Spark + Kafka): Built a streaming pipeline to consume Kafka topics, process messages in real-time, and visualize outputs.

Deployed using Docker and Jupyter for reproducibility.

Tumult Analytics is a Python library for privately computing aggregate queries on tabular data. It is built atop the Tumult Core library.

Introduction to PySpark

Deploying PySpark Jobs on Azure HDInsight Spark Cluster (CI/CD)

Funnel provide an easy to use, easy to read framework to create very complex data selections over pandas DataFrames

"Cloud container data analytics, statistical modeling, and machine learning on distributed databases". "A free opensource alternative to SPSS, SAS, MATLAB, PowerBI, Tableau and Alteryx". Runs on Linux, Windows, MacOS, and in the cloud via containers.

Mirror of https://github.com/zero323/pyspark-asyncactions

Yelp open dataset explorer using spark and cassandra