K

kafka

Hybrid Filtering with Spark MLlib

A living error knowledge base for the Nielsen Fulfillment Center (NFC) — Realtime bugs logged, diagnosed, and fixed across the full stack (Java Spring Boot, React, MAF-CLI, PostgreSQL, Kafka). Built on the 0xDEBUG universal debugging taxonomy: a multiple-file Obsidian knowledge graph and Gemini AI assistant that classifies any error into one of the failure patterns across various technical domains.

1000+ DevOps Bash Scripts - AWS, GCP, Kubernetes, Docker, CI/CD, APIs, SQL, PostgreSQL, MySQL, Hive, Impala, Kafka, Hadoop, Jenkins, GitHub, GitLab, BitBucket, Azure DevOps, TeamCity, Spotify, MP3, LDAP, Code/Build Linting, pkg mgmt for Linux, Mac, Python, Perl, Ruby, NodeJS, Golang, Advanced dotfiles: .bashrc, .vimrc, .gitconfig, .screenrc, tmux..

Sistema modular y orientado a eventos para gestionar artistas, usuarios, conciertos, reservas con caducidad y pagos mediante Stripe o un emulador local. Construido con Java 25, Spring Boot 4.0.6, Spring Cloud, Kafka y PostgreSQL, siguiendo DDD y Arquitectura Hexagonal. Incluye seguridad OAuth2/JWT y notificaciones en tiempo real mediante SSE.

Modular, event-driven system for managing artists, users, concerts, expiring reservations, and payments through Stripe or a local emulator. Built with Java 25, Spring Boot 4.0.6, Spring Cloud, Kafka, and PostgreSQL, following DDD and Hexagonal Architecture. Includes OAuth2/JWT security and real-time notifications via SSE.

Change data capture on Oracle databases + transfer of change events to Kafka

Java microservice template built with Spring Boot and Apache Kafka, following Event-Driven Architecture (EDA). Provides a modular, scalable foundation for asynchronous distributed systems.

Kafka from install to advanced: ZooKeeper, KRaft, Redpanda console, partitions, consumer groups, monitoring, SSL, ZooKeeper migration. Xavki tutorials — https://youtube.com/playlist?list=PLn6POgpklwWreX6DssE1tdt9O5rJe0naK

Captures SAP changes, transforms them into normalized events, and publishes to Kafka for processing.

WebSocket proxy that allows clients to produce and consume messages from topics in Apache Kafka. Documentation can be found at https://kpmeen.gitlab.io/kafka-websocket-proxy/

DataRider IoT Stream Processing (IT Module) in Python with Kafka

A robust step for IoT data ingestion, designed to read serial data and stream it directly to Kafka. This module is part of the DataRider ecosystem, focusing on seamless data transfer from IoT devices to cloud or on-premise data pipelines.

Notes

This repository is a fork of our internal GitLab instance, utilizing JFrog Artifactory for container registry management.

Continuous Integration (CI) pipelines are configured to run exclusively within our private GitLab environment.

Lab CDC event-driven : PostgreSQL → Debezium → Kafka → consumers Python (notifications, audit, risk scoring). Illustre WAL, replication slots, outbox pattern et eventual consistency sur un cas bancaire réel.

Krun reads events from a Kafka topic and run a system command passing the read event to the stdin of the command. The command can be a bash script or anything else that can manage a stdin.

Streaming data pipeline for European Day-Ahead electricity market — Cats Effect 3 + fs2 + http4s + Kafka

Learning Spark in java/scala

Projet de référence d'une architecture Lakehouse moderne appliquée à la détection de fraude bancaire.

Simule un environnement de production avec trois sources de données hétérogènes (fichiers CSV, base PostgreSQL, streaming Kafka/Redpanda) ingérées en continu vers un stockage objet S3-compatible (MinIO).

Stack technique :

Ingestion batch : Apache Spark (PySpark) + Delta Lake Ingestion streaming : Spark Structured Streaming + Redpanda (Kafka) Orchestration : Apache Airflow Transformation : dbt (DuckDB) Stockage : MinIO (S3), Delta Lake (Bronze/Silver), Parquet (Gold) Exploration : DuckDB / DBeaver

Architecture en médaillon (Medallion Architecture) :

Bronze : données brutes, sources séparées Silver : données nettoyées, déduplication inter-sources Gold : agrégats métier (fraude par heure)

L'ensemble de la stack tourne en local via Docker Compose.

Production-style real-time crypto streaming pipeline (Binance → Kafka → Polars Parquet → DuckDB → dbt) built locally with Docker and Python.

Real-time stock market data lakehouse using Kafka, Bronze/Silver/Gold architecture, and Postgres feature serving.

Real-time connected vehicle telemetry pipeline using Kafka, MongoDB, FastAPI, and Grafana with anomaly detection and fleet monitoring dashboards.

-> Contenu -Un seul topic order-events sur lequel sont publiés trois types d’événements : OrderCreated, PaymentValidated, OrderShipped. -Producer (REST API) : publie les événements ; Consumer : écoute le topic avec trois consumer groups (groupIds distincts) et délègue au domaine via des handlers. -Architecture : hexagonale + template method ; contrats partagés dans le module kafka-contracts (OpenAPI). -> Stack: -Java 21, Spring Boot 3.2, Spring Kafka -Kafka (Zookeeper), Kafka UI -Maven multi-module : kafka-contracts, order-event-producer, order-event-consumer -Docker / Docker Compose pour l’infra et les apps -CI/CD GitLab : build Maven, tests unitaires, build & push des images sur Docker Hub, déploiement par SSH + docker compose sur le serveur. -> Dépôts d’images -Images poussées sur Docker Hub par la CI : kafka-kata-producer, kafka-kata-consumer.

Sistema event-driven con Kafka que transforma documentos no estructurados en especificaciones de software completas. Extrae texto con OCR, procesa NER con transformers, clasifica oraciones y generar SRS en múltiples formatos.