<!--- Please read this! Before opening a new issue, make sure to search for keywords in the issues filtered by the "regression" or "bug" label: - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=bug and verify the issue you're about to submit isn't a duplicate. ---> ### Summary The pipeline build starts, but the 'running' status is not updated after all the jobs completed on the MR Pipelines tab. ### Steps to reproduce No reproduction scenario, because most of the time the status is correct. ### What is the current *bug* behavior? The status of the build is not correct: ### What is the expected *correct* behavior? When all builds finished in a pipeline, the pipeline should not show the status 'running' (What you should see instead) ### Relevant logs and/or screenshots  See examples also in this [comment](https://gitlab.com/gitlab-org/gitlab/-/issues/36237#note_922427849) ### Output of checks This bug happens on our own hosted gitlab (12.3.6-ee) #### Results of GitLab environment info <details> <summary>Expand for output related to GitLab environment info</summary> <pre> (For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`) </pre> </details> #### Results of GitLab application Check <details> <summary>Expand for output related to the GitLab application check</summary> <pre> (For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:check SANITIZE=true`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true`) (we will only investigate if the tests are passing) </pre> </details> ### Possible fixes Self-Heal jobs in the background if they have been active for a while. Cron worker to batch through old alive pipelines and run PipelineProcessWorker on them if the they are in an inconsistent state(jobs are stopped). ```ruby class MyWorker # Do we limit this to older pipelines like `Ci::StuckBuilds::Drop*Worker` Ci::Pipeline.where(status: [Ci::HasStatus::ACTIVE_STATUSES]).each do |pipeline| jobs_stopped = CommitStatus.where(pipeline: pipeline) .pluck(:status) .all? { |status| Ci::HasStatus::STOPPED_STATUSES.include?(status) } if pipeline.need_processing? && jobs_stopped Ci::ProcessPipelineService .new(pipeline) .execute end end end ```