Update pipeline / stage / build statuses in an atomic transaction when build status changes
We recently moved some parts of pipeline processing process to Sidekiq, what means that we now process pipeline asynchronously.
This helps to avoid some performance problems, some race conditions, but at the same time, it introduces another problems, and another race conditions.
One problem, that may be visible to users, is changing the status of the build. When build changes status we have to update pipeline status as well. We do that by triggering update_status
action on pipeline. Currently, after recent changes, this happens asynchronously too, but this may be quite unexpected, when someone retries build (thus changes status of the build and the entire pipeline). But pipeline status will be changed after Sidekiq picks this job, so when someone visits pipelines pages in the meantime, he will still see the old status.
I think it may be a good idea to update pipeline status synchronously (despite performance concerns), so we should probably revert that change. What do you think @ayufan?
EDIT: After we introduced persisted stages, we need to update pipeline <- stage <- build status at the same time. There are some solutions of this problem described in the issue discussion.
Current proposal is described in https://gitlab.com/gitlab-org/gitlab-ce/issues/23257#note_91487708