Speed up pipeline creation for gitlab-org/gitlab
Problem
Pipeline creation is slowing down with the increasing number of jobs and rules that are being added and have to be parsed/evaluated. There are no low hanging fixes left that we can make to speed it up: &7290
Web requests often timeout when the pipeline creation takes too long.
The fastest way to get results and improve dev efficiency is to reduce the number of jobs that are created in an HTTP request.
Proposal
General solution
Moved to: #463355
Parent-child pipeline workaround
Child pipelines are created async
Leverage parent-child pipelines for gitlab-com/gitlab
by moving the entire current pipeline to be executed as a child pipeline.
Create a new .gitlab-ci-parent.yml
custom configuration file containing a trigger for the current pipeline and the current workflow rules:
# .gitlab-ci-parent.yml
create_pipeline:
trigger:
include:
- local: .gitlab-ci.yml
strategy: depend
rules:
- if: '$CI_MERGE_REQUEST_LABELS =~ /pipeline:create-async/'
include:
- local: .gitlab-ci.yml
rules:
- if: '$CI_MERGE_REQUEST_LABELS == /pipeline:create-async/'
when: never
This way we can run it as an experiment without having to change the .gitlab-ci.yml
file and could be opt in for the duration of the test.
Pros:
- the parent pipeline is created almost instantly
- the child is created async in Sidekiq
- the status is propagated from the child to the parent, so the time from
run new pipeline
to settingmwps
is reduced since we have a running pipeline as soon as the button is clicked.
Cons:
- it is more complex to navigate, it require a couple more clicks
- If we take this approach it's more of a band-aid. Customer's won't benefit from the work, only us.