Pipeline and Job creation is too intertwined with our GitLab CI YAML configuration

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Problem

Several of the pitches for the Verify hackathon involved structural changes in how pipeline are configured and executed. Today, these changes are very difficult to make because the creation of Pipeline and Build objects, to in turn be saved to the database, and very tightly coupled to the actual YAML used to configure the pipeline. Today:

  1. We cannot create a pipeline without also creating all of the pipelines jobs. This means that we must do all job-specific configuration evaluation before the pipeline is created, and configure jobs based on the behavior of other jobs in the pipeline. verify-stage#203 (comment 956921725)
  2. We cannot add a job to an existing pipeline. verify-stage#203 (comment 975917115)
  3. We cannot use actor-based feature flags to CI configuration features verify-stage#203 (comment 937835172)

Proposal

I propose that we create and expose a simple and well-structured Ruby API that defines what information is required to create Ci::Pipeline and Ci::Build. Roughly, we'd pass it something like a hash of pipelines attributes , which includes a list of hashes that each represent build, etc. More specifically, they'd probably represent a Ci::Pipeline::Seed::Pipeline and a list of Ci::Pipeline::Seed::Build. But in doing this, we'll allow CI configuration to be completely modular. We would be able to:

  1. Use a JSON Schema to validate configuration that we'll know, in turn, can create valid Pipelines and Jobs verify-stage#203 (comment 988081127)
  2. Configure a Pipeline full of Jobs with Cue, if we feel like it verify-stage#203 (comment 952811770). We could develop and use any number of alternative configuration processes, so long as the produced data structure adheres to the known Pipeline and Job schema.

By freeing up the YamlProcessor from the Pipeline creation process, we would then be able to:

  1. Package the YamlProcessor as a gem, which can be properly versioned #222522
  2. Use actor-based feature flags in the application, where CI processing logic lives, to decide what CI functionality we want to enable independent of the configuration language
  3. Use versioning and initialization-parameter based toggling for enabling specific YAML configuration features independent of CI execution
Edited by 🤖 GitLab Bot 🤖