How to programmatically test CI configurations
[This is mostly a discussion issue. Feature proposal issues might be created out of it]
For projects with very complex CI configuration (omnibus-gitlab
, for example), doing refactors on the CI configuration is a herculean task. It boils down to creating multiple forks of the project and then doing the testing their and manually verifying all the pipelines act as expected after the changes are applied. You can take a look at https://docs.gitlab.com/omnibus/development/pipelines.html to see all the possible combinations of pipelines and jobs that omnibus-gitlab
contains. Add to it certain jobs being run/skipped based on environment variable values, the mess becomes more complex.
What could help here is if we had public method that would take in the following inputs (not an exhaustive list), and spit out a JSON output of the pipeline that will be created:
- CI configuration raw text.
- Project details - (path, name, etc. or their
CI_*
variable counterparts). - How the pipeline is created -
source
(push or trigger or schedule or multi-project pipeline etc.) - Against which
ref
the pipeline is being run. - Extra environment variables that might be specified via Project/Group settings or new pipeline form or git push options.
For example, the method should be able to answer a question like "How would the pipeline created by this CI configuration on a project with path gitlab-org/omnibus-gitlab
with variables ee=true
against the ref 1.2.3+456
look like? What all jobs will it have?"
Right now, we have "simulate pipeline against default branch" which does a very minimal version of this. But that is very limited and can't be used to cover all possible options.
@drew rightly pointed me towards Ci::CreatePipelineService#execute
method which might be able to help us with this. The question is, how can we make this a public thing that can be used by other projects too?
/cc @grzesiek @steveazz since you also were interested in it.