⚡️ Faster Pipeline - LAB-483

Pipeline Optimization

While I was waiting for some evening MRs to be merged, I noticed that the pipeline was rather slow so I had a closer look 👀

First of all, I noticed that wether we needed OWL and JSON-LD files or not, they were generated alongside Shacl files, so I started by separating those jobs in a single job per file type. This uses more resources, which isn't a big issue as those jobs are only run when merging to develop or main, but there are benefits.

Faster Pipeline

MR pipelines are now faster because we only generate the Shacl files instead of all file types. This are divided by 2 and that's a good thing for the feedback loop.

Before :

After :

Parallel Execution

Another benefit of this pipeline model is that builds are now considered a directed acyclic graph instead of sequential stages/jobs.

This means that jobs can be run in parallel on different Gitlab Runners, a typical MR pipeline will now run the Javascript tests and Python instance validation tests at the same time hence making the overall process quicker.

What's even better is that develop pipelines can run all the file generation jobs at the same time and allow downstream jobs to run before all the file types are generated, for example Shacl tests can be executed before JSON-LD and OWL files are generated with the new dependency model.

Easier Debugging

To date, if a generator has an issue during the process, it is impossible to pinpoint which generator failed (JSON-LD ? Shacl ? OWL ?). With the new pipeline model, if a generator fails, a red cross will appear next to the generator's name.