PipelineProcessService is executed a multiple times.
This results in a quite an overhead on the amount of compute that it uses.
We should (my random notes of improvements of this service, we can do a lot to make it take likely 10% of current time).
- Debug all SQL queries being executed,
- De-duplicate the jobs being executed, being done by https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/31370,
- Make pipeline processing more efficient,
- Serialise updates created by processing of individual jobs during the
- Make updates of stages to be targeted to builds being triggered by
- Ignore update of pipeline if build was retried,
- Be aware of
when:to update only builds that are in fact affected,
- Remove any potential
N+1, try to pre-calculate
SQLonly, or be clever to discover which builds that are "created" should be processed,
- We use
pipeline.builds.to gather status of prior stages, this seems to be bug as we should be using
pipeline.statuses(to also include bridges),
- Ensure that we always use
- Ensure that update of
stagesare sequential across all concurrent runs,
- Remove deprecated code for
update_retriedas we no longer need it,