Improve `PipelineProcessService`
Summary
Currently PipelineProcessService
is executed a multiple times.
This results in a quite an overhead on the amount of compute that it uses.
We should (my random notes of improvements of this service, we can do a lot to make it take likely 10% of current time).
- Debug all SQL queries being executed,
- De-duplicate the jobs being executed, being done by https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/31370,
- Make pipeline processing more efficient,
- Serialise updates created by processing of individual jobs during the
process!
, - Make updates of stages to be targeted to builds being triggered by
process!
, - Ignore update of pipeline if build was retried,
- Be aware of
when:
to update only builds that are in fact affected, - Remove any potential
N+1
, try to pre-calculatestatus
inSQL
only, or be clever to discover which builds that are "created" should be processed, - We use
pipeline.builds.
to gather status of prior stages, this seems to be bug as we should be usingpipeline.statuses
(to also include bridges), - Ensure that we always use
.find_each
, - Ensure that update of
stages
are sequential across all concurrent runs, - Remove
rubocop
offenses, - Remove deprecated code for
update_retried
as we no longer need it,