Pre-pipelines

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Problem to solve

We have multiple topics, especially in Secure, where having a detection layer before running the pipeline would be very useful.

Intended users

Further details

I'd like to challenge the idea of having a "pre" pipeline, running before any other job, and allowing the rest of the pipeline to be shaped by the result of this preliminary step. This would be the equivalent of before_script but at the pipeline level, let's call it "before_job" to balance with it.

If we had such job, we (~"devops::secure") could create advanced detection logic, in order to make our Security Products easier to use, and more performant. We could detect for example the exact version of interpreters of frameworks, and trigger some jobs instead of others (like a Java 13 instead of the current default Java 8 for spotbugs).

Proposal

The result of this preliminary job(s) would be a file containing key-value pairs like the output of env. The same way we define variables in GitLab-CI, but this time the values are the result of a job execution.

The pipeline would technical divided in two: the before_job itself, and the rest of the pipeline that would be scheduled only after evaluating this file. This means we would run pipelines in two steps instead of one. And that's a fairly heavy change to consider, but we don't need to expose this complexity to users (ie: We need only one id for the 2 parts of the pipeline). Also, if the "before_job" fails, the whole pipeline would stop by default.

I'm not sure how we could have multiple jobs before running the regular pipeline, because we can' really define an order, so maybe it should only be one job. We need to figure out also the precedence of the variables in the generated file.

Permissions and Security

No change.

Documentation

Update https://docs.gitlab.com/ee/ci/yaml/#variables

Availability & Testing

TBD.

What does success look like, and how can we measure that?

  • Number of pipelines using a "before_job"

What is the type of buyer?

GitLab Ultimate?

Is this a cross-stage feature?

Probably, but I can't see other stages than Verify for now.

Links / references

Edited by 🤖 GitLab Bot 🤖