GitLab CI Events
Disclaimer: The following contains information related to upcoming products, features, and functionality.
It is important to note that the information presented is for informational purposes only. Please do not rely on this information for purchasing or planning purposes.
As with all projects, the items mentioned in this document and linked pages are subject to change or delay. The development, release and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.
Current Status
The PoC is now done! It can be found here: !91244 (closed). Demo (available only for GitLab team members): https://youtu.be/cwfRI9m3rRs.
Problems to Solve
- Multiple pipeline definitions in a single project
- Ability to define triggers for such pipelines where a trigger can be virtually any event within the system (ex. an issue is created, label assigned, container pushed)
- Ability to compose (third-party) steps together to form a job, a prerequisite to building an ecosystem like the GitHub Actions Marketplace
Proposed PoC / MVP details for GitLab CI Workflows
How to build GitLab CI Workflows in an efficient way, making use of what we already have? There are many ways to implement this, we could build a separate decoupled service to dispatch workflows, but in the PoC we can actually use existing hooks around webhooks to dispatch events-to-builds.
- Make it possible to define a workflow in
.gitlab-ci.yml
- Preserve this definition in database and connect to existing webhooks.
- Hook into existing webhooks instrumentation to start a pipeline when a workflow definition matches.
- Write file-type
CI_WORKFLOW_EVENT_PAYLOAD
variable to pipeline builds with the webhook payload. - Add a sub-page of pipelines / job to visualize workflows better.
- Add a system note to a writable resource (issue / merge request) when a workflow executes.
A few implementation details
Configuration
Workflow definition in .gitlab-ci.yml
can reuse existing workflow
keyword. Perhaps we could add support for something like this:
workflow:
rules:
- # ...
events:
my-workflow-1:
issues:
- open
- close
my-job:
needs: my-workflow-1
script: run-my-script
This way we separate workflow definitions from jobs and pipelines, working around having only single CI configuration file. In case of GitHub you can have multiple workflow YAMLs, making it easier to define different workflow with top-level on:
configuration.
Initially we would add support for all webhook events as we do not have too many. This would immediately unlock tremendous value. GitHub has much more events than we have in webhooks today.
Persistence
Create ci_workflow_events
and ci_workflow_runs
database tables. In the PoC / MVP only support project-level workflows, as it might be very expensive to look up workflows in every webhook execution in a larger scope.
Store workflow definition along with project_id
in ci_workflow_events
, match a workflow upon webhook firing (most of the code is in the codebase already).
In ci_workflow_runs
store a one-to-many relationship between ci_workflow_events
and ci_pipelines
with additional metadata around a workflow run.
Run
When a workflow matches create a new "workflow-type" pipeline for a given project. Dispatching events / pipelines could be a fairly expensive, as we would need to match a workflow whenever we fire a webhook. In order to reduce the impact we can only support project-level workflows in the MVP what should make performance acceptable initially.
Once a pipeline starts, all jobs defined with needs: workflow-configured-name
will receive CI_WORKFLOW_EVENT_PAYLOAD
file-type variable pointing to a file filled with a corresponding webhook payload in JSON format. Users will need to use tools like jq
to consume it, but for MVP it might be sufficient.
All the pipelines with workflow-jobs will be visualized in the Pipelines / Workflows
page. A job details page will contain details about workflow that triggered it and its definition.
When a workflow is triggered by a merge request or issue, add a system note and link to a triggered pipeline.
Scaling
This design may not scale well, but will allow us to experiment with MVP in a limited availability of this feature. As adoption grows we may need to update the design to match the scale in which this feature will operate in. Later we may want to disconnect it from webhooks or to generalize webhooks to be a foundation of the next design of this feature. From the MVP we would be able to iterate towards making this feature generally available!
I know that @ayufan might be interested in making this generally available and / or contributing to the initial design, so we may be able to get some support from Kamil to make sure that MVP is successful.