Proposal to move pipeline creation to Pipeline Authoring

Background

Today many of the Verify endpoints fall under continuous_integration feature category, which is owned by grouppipeline execution.

The table below shows the number of occurrences where we specify feature_category in code. This is just rough data. In reality continuous_integration category covers many more endpoints because we tend to bulk assign endpoints with feature_category at the top of the file (e.g. for a whole Controller class having multiple endpoints/actions).

feature count
continuous_integration grouppipeline execution 76
build_artifacts grouppipeline security 17
code_testing grouppipeline execution 13
secrets_management grouppipeline security 11
pipeline_composition grouppipeline authoring 10
merge_trains grouppipeline execution 2
review_apps grouppipeline execution 1

Problem

grouppipeline authoring owns most of the logic in Ci::CreatePipelineService since the majority of it is about building a pipeline from CI configurations. In fact grouppipeline authoring owns the CI Linter endpoint. The Linter is logic used in great part by Ci::CreatePipelineService. Yet, grouppipeline execution is responsible for the error budget of all pipeline creation endpoints.

Effort on error budget is heavily skewed towards grouppipeline execution owning the majority of critical endpoints:

Proposal

Assign feature_category :pipeline_composition to all endpoints responsible for the pipeline creation:

  • RunPipelineScheduleWorker
  • PipelinesController#create
  • CreatePipelineWorker
  • Ci::ExternalPullRequests::CreatePipelineWorker
  • MergeRequests::CreatePipelineWorker
  • POST /trigger/pipeline
  • POST /pipeline
  • POST /merge_requests/:mr_id/pipelines

grouppipeline authoring covers pipeline_composition. Arguably, assembling a pipeline (creation) is also pipeline_composition.

Challenges

  1. Some of these endpoints are coupled to other features owned by grouppipeline execution. For example Ci::ExternalPullRequests::CreatePipelineWorker. Arguably, this coupling is minimal and the primary responsibility of those endpoints is to create a pipeline. They usually carry minimal contexts of other features (e.g. merge requests, external pull requests, triggers, etc.) and they primarily call Ci::CreatePipelineService.
Edited by Fabio Pitino