Update February 22nd 2022
All Weekly Demos: #16
Recording
Vision
Make GitLab a tool Data Scientists and Machine Learning Engineers love to use.
Mission
Explore and Collaborate with different teams to deliver features that improve the user experience for Data Scientists and Machine Learning Engineers, while increasing awareness within the company to this user groups
What Was Done
Gitlab Pipelines for Data Science use cases
https://gitlab.com/gitlab-org/incubation-engineering/mlops/poc/citer
Pipeline are a key part of the Data Science and Machine Learning workflow, and differently from regular software they are already important on the Create stage. Therefore, we are exploring a way of creating, iterating and running pipelines without relying on git, and we created a POC that achieves this: Citer. Citer uses a pre-configured pipeline at is triggered by api, and accepts a CI configuration, variables and extra files needed to run that CI file, without the need for them to live in the repository.
So, instead of the change -> commit -> wait for git hooks -> push -> open gitlab -> search pipeline -> wait for ui to update -> check results -> repeat we now have change -> trigger pipeline -> wait for pipeline to complete -> check results -> repeat, which greatly improves productivity when creating or changing a pipeline. Once fetching job results is added, the entire process can be done without opening GitLab.
Now, this is clearly a POC meant to showcase the experience, but we do not recommend its usage, since it relies on APIs that are not really meant for this. For example, it uses the issues API to upload the extra files necessary, and using the Dynamic Parent Child pipelines still adds a lot of pipeline overhead. We will be creating a video explaining the technical implementation along the week.
Jupyter Rendered Diffs
gitlab-org/gitlab!75500 (merged)
Swapping between clean and rendered diffs is being reviewed and will hit version 14.9, will start working on adding this to the MR page (gitlab-org/gitlab#352872)
Up Next
We will continue working on the MR for Jupyter Rendered Diffs, while adding efforts into pipelines.