D

Data Engineering

my solutions and materials from courses on https://karpov.courses

ELT pipeline orchestrated with Apache Airflow, extracting YouTube data via the YouTube Data API v3, loading it into PostgreSQL (staging & core schemas), with data quality checks using Soda. Containerized with Docker Compose, tested with Pytest, and automated via GitLab CI/CD.

This repository holds code for the AskAnna Backend. Our backend stack primarily uses Django and the Django REST Framework.

Cloud-native data engineering + ML POC: ingest Reddit images, run OCR, store results in BigQuery/Cloud Storage, and serve analytics via FastAPI + a React dashboard.

Collection of containers and orchestration scripts that deploy a complete environment for developing Data Science projects. Developed based on my current needs in my current role.

Live data source that can be used for data engineering, data warehouse and etl development.

Live data source that can be used for data engineering, data warehouse and etl development.

Live data source that can be used for data engineering, data warehouse and etl development.