D

Data Engineering

Set up Power BI incremental refresh in minutes, without admin approval, production access or a single line of code. 👀

So, we’ve been tinkering, tankering and tunkering, and we have our latest release. 🚀 Power BI Incremental Refresh is a free, open source tool built for anyone who has ever babysat a slow refresh.

💻 Work locally, no sign in needed. Open a Power BI Project (.pbip) straight from your computer and set up incremental refresh in a few clicks. Your files never leave your browser. Download the updated model, test it in Desktop and commit it to git.

🔧 Or work directly in the Power BI service. Pick a published model, answer a few questions, and it adds the parameters, filters and refresh policy for you. No Tabular Editor, no XMLA scripts. You see every change line by line before it’s applied, with a backup and one click undo.

🩺 Health check a whole workspace. It scans every semantic model and flags: • Tables quietly loading only one month of data because a policy was lost on republish • Policies that duplicate rows • Filters that can’t fold to the source, so every refresh still reads everything • Failed and slow refreshes creeping towards the time limits • Big fact tables that should be using incremental refresh but aren’t

Every problem comes with a one click fix, and you can download the results as a report for your team.

🔒 Security reviewed, with the results in the README. There’s no backend and no data stored, it acts with your own Power BI permissions, and nothing is loaded from third party sites.

👉 Try it now: https://pbi-incremental-17fe26.gitlab.io/ Open your own .pbip project, or click “Demo the health check” to see it find problems in seconds.

I’d love to hear from anyone working with Power BI or Fabric. What’s the most painful refresh you’ve had to babysit? 👇

Local environment setup using Kafka, ksqlDB, AKHQ, and Schema Registry. Ready-to-use, Docker-based configurations and scripts to run the environment for data engineering workflows, microservices, and ETL pipelines.

my solutions and materials from courses on https://karpov.courses

Fundamental theory and practice in Data Science (DS). 🧮

ELT pipeline orchestrated with Apache Airflow, extracting YouTube data via the YouTube Data API v3, loading it into PostgreSQL (staging & core schemas), with data quality checks using Soda. Containerized with Docker Compose, tested with Pytest, and automated via GitLab CI/CD.

This repository holds code for the AskAnna Backend. Our backend stack primarily uses Django and the Django REST Framework.

Cloud-native data engineering + ML POC: ingest Reddit images, run OCR, store results in BigQuery/Cloud Storage, and serve analytics via FastAPI + a React dashboard.

tools for processing semantic data packaged as Nextflow modules

This GitLab project is merely for displaying the actual project, which I did without GitLab, here in my portfolio. It contains a short README about the goals and challenges of the project and how I solved it.

Risk segmentation and premium optimization for car insurance in South Africa using statistical analysis and machine learning for AlphaCare Insurance Solutions.

This pet project is part of my personal journey in Data Engineering (DE) to explore new data stacks and various possibilities as a Data Warehouse developer. I would appreciate it if you could raise issues and help me understand certain aspects.

Collection of containers and orchestration scripts that deploy a complete environment for developing Data Science projects. Developed based on my current needs in my current role.

Quick and dirty pipeline to prepare a dataset for mobility studies, starting from a collection of CDRs.

Funnel provide an easy to use, easy to read framework to create very complex data selections over pandas DataFrames

Live data source that can be used for data engineering, data warehouse and etl development.

Live data source that can be used for data engineering, data warehouse and etl development.

Live data source that can be used for data engineering, data warehouse and etl development.