ci: SSOT to determine a CI/CD pipeline type from the jobs it is running
Context
We now determine a CI/CD pipeline type based on the job it runs in three locations:
- The
scripts/pipeline/set-pipeline-name
script - SQL queries in Snowflake
- The
scripts/pipelines-select
script
Goal
Ideally have one location where we set the pipeline type, and reuse this type in other places.
Technical ideas
Setting the type as part of the pipeline name in scripts/pipeline/set-pipeline-name
in the gitlab-org/gitlab
CI/CD pipeline, and reuse it in Snowflake and other scripts, sounds like a promising approach. It has a few drawbacks:
- Update previous CI/CD pipelines that don't have a type in their name: assume we roll this out soon, and use the pipeline name in Snowflake. We would need to run a "migration" to rename all pipelines in the last year or so to be able to have historical data (we currently compute the pipeline type every time a query is run)
- Update pipeline type after every change: If after the name is added to pipelines, we want to change the naming logic (e.g. a pipeline has the
rspec-predictive
main type if it has the jobdef
, and previously it used to be when it had the jobabc
), older pipelines with the typerspec-predictive
would be wrong by the new definition, and we would need to change the name for all pipelines.
We run around 7'000 MR pipelines/month, so that would be 84'000 MR pipelines to change every time we make a change in our CI/CD job names.
Edited by David Dieulivol