_Disclaimer: The following contains information related to upcoming products, features, and functionality._ _It is important to note that the information presented is for informational purposes only. Please do not rely on this information for purchasing or planning purposes._ _As with all projects, the items mentioned in this document and linked pages are subject to change or delay. The development, release and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc._ ## Current Status The PoC is now done! It can be found here: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91244. Demo (available only for GitLab team members): https://youtu.be/cwfRI9m3rRs. ## Problems to Solve 1. Multiple pipeline definitions in a single project 2. Ability to define triggers for such pipelines where a trigger can be virtually any event within the system (ex. an issue is created, label assigned, container pushed) 3. Ability to compose (third-party) steps together to form a job, a prerequisite to building an ecosystem like the GitHub Actions Marketplace 4. Use cases to solve 1. https://gitlab.com/gitlab-org/gitlab/-/issues/215502#note_1905673788 ## Proposed PoC / MVP details for GitLab CI Workflows How to build GitLab CI Workflows in an efficient way, making use of what we already have? There are many ways to implement this, we could build a separate decoupled service to dispatch workflows, but in the PoC we can actually use existing hooks around webhooks to dispatch events-to-builds. 1. Make it possible to define a workflow in `.gitlab-ci.yml` 2. Preserve this definition in database and connect to existing webhooks. 3. Hook into existing webhooks instrumentation to start a pipeline when a workflow definition matches. 4. Write file-type `CI_WORKFLOW_EVENT_PAYLOAD` variable to pipeline builds with the webhook payload. 5. Add a sub-page of pipelines / job to visualize workflows better. 6. Add a system note to a writable resource (issue / merge request) when a workflow executes. ## A few implementation details #### Configuration Workflow definition in `.gitlab-ci.yml` can reuse existing `workflow` keyword. Perhaps we could add support for something like this: ``` workflow: rules: - # ... events: my-workflow-1: issues: - open - close my-job: needs: my-workflow-1 script: run-my-script ``` This way we separate workflow definitions from jobs and pipelines, working around having only single CI configuration file. In case of GitHub you can have multiple workflow YAMLs, making it easier to define different workflow with top-level `on:` configuration. Initially we would add support for all [webhook events](https://docs.gitlab.com/ee/user/project/integrations/webhook_events.html) as we do not have too many. This would immediately unlock tremendous value. GitHub has much more events than we have in webhooks today. #### Persistence Create `ci_workflow_events` and `ci_workflow_runs` database tables. In the PoC / MVP only support project-level workflows, as it might be very expensive to look up workflows in every webhook execution in a larger scope. Store workflow definition along with `project_id` in `ci_workflow_events`, match a workflow upon webhook firing (most of the code is in the codebase already). In `ci_workflow_runs` store a one-to-many relationship between `ci_workflow_events` and `ci_pipelines` with additional metadata around a workflow run. #### Run When a workflow matches create a new "workflow-type" pipeline for a given project. Dispatching events / pipelines could be a fairly expensive, as we would need to match a workflow whenever we fire a webhook. In order to reduce the impact we can only support project-level workflows in the MVP what should make performance acceptable initially. Once a pipeline starts, all jobs defined with `needs: workflow-configured-name` will receive `CI_WORKFLOW_EVENT_PAYLOAD` file-type variable pointing to a file filled with a corresponding webhook payload in JSON format. Users will need to use tools like `jq` to consume it, but for MVP it might be sufficient. All the pipelines with workflow-jobs will be visualized in the `Pipelines / Workflows` page. A job details page will contain details about workflow that triggered it and its definition. When a workflow is triggered by a merge request or issue, add a system note and link to a triggered pipeline. #### Scaling This design may not scale well, but will allow us to experiment with MVP in a limited availability of this feature. As adoption grows we may need to update the design to match the scale in which this feature will operate in. Later we may want to disconnect it from webhooks or to generalize webhooks to be a foundation of the next design of this feature. From the MVP we would be able to iterate towards making this feature generally available! :rocket: I know that @ayufan might be interested in making this generally available and / or contributing to the initial design, so we may be able to get some support from Kamil to make sure that MVP is successful.