pipeline shows running when all jobs finished

Summary

The pipeline build starts, but the 'running' status is not updated after all the jobs completed on the MR Pipelines tab.

Steps to reproduce

No reproduction scenario, because most of the time the status is correct.

What is the current bug behavior?

The status of the build is not correct:

What is the expected correct behavior?

When all builds finished in a pipeline, the pipeline should not show the status 'running' (What you should see instead)

Relevant logs and/or screenshots

See examples also in this comment

Output of checks

This bug happens on our own hosted gitlab (12.3.6-ee)

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:env:info)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Possible fixes

Self-Heal jobs in the background if they have been active for a while.

Cron worker to batch through old alive pipelines and run PipelineProcessWorker on them if the they are in an inconsistent state(jobs are stopped).

class MyWorker
  # Do we limit this to older pipelines like `Ci::StuckBuilds::Drop*Worker`
  Ci::Pipeline.where(status: [Ci::HasStatus::ACTIVE_STATUSES]).each do |pipeline|
    jobs_stopped = CommitStatus.where(pipeline: pipeline)
      .pluck(:status)
      .all? { |status| Ci::HasStatus::STOPPED_STATUSES.include?(status) }

    if pipeline.need_processing? && jobs_stopped
       Ci::ProcessPipelineService
        .new(pipeline)
        .execute
    end
  end
end