Could not retrieve the pipeline status

Summary

The MR widget shows Could not retrieve the pipeline status instead of pipeline status. But the pipeline tab in the widget shows the pipelines without issue.

Steps to reproduce

Unable to reproduce this issue, because I don't know what's causing things to get into this state.

Example Project

This MR is closed and merged, but it should show the same behavior: cody/auto-devops-templates!1 (merged). I'm not sure how this MR got into this state, as I only accidentally reproduced the bug.

What is the current bug behavior?

The group/project/merge_requests/:id/widget.json endpoint returns no pipeline object. I had a chance to trace through the code and test some things with a self-managed customer that was experiencing this behavior. I think this is happening because actual_head_pipeline is returning nil for some merge requests.

I can see in serializers/merge_request_poll_widget_entity.rb#L20 that :actual_head_pipeline is what gets exposed as :pipeline.

I think this might be a regression that was introduced with gitlab-foss#40004 (closed).

Running update_head_pipeline fixes the issue.

What is the expected correct behavior?

Pipeline status is returned and displayed in the MR widget.

Relevant logs and/or screenshots

Here's the text from a rails console session where we were testing some things:

irb(main):003:0> mr = MergeRequest.find_by(project: proj, iid: 154)
=> #<MergeRequest id:217665 group/project!154>
irb(main):004:0> mr.attributes
=> {"id"=>217665, "target_branch"=>"master", "source_branch"=>"something-bugfix/other-words-here", "source_project_id"=>21708, "author_id"=>9986, "assignee_id"=>nil, "title"=>"MR title here", "created_at"=>Fri, 01 Nov 2019 22:07:07 UTC +00:00, "updated_at"=>Mon, 04 Nov 2019 18:37:59 UTC +00:00, "milestone_id"=>nil, "state"=>"closed", "merge_status"=>"can_be_merged", "target_project_id"=>21708, "iid"=>154, "description"=>"description here", "updated_by_id"=>nil, "merge_error"=>nil, "merge_params"=>{"force_remove_source_branch"=>"0"}, "merge_when_pipeline_succeeds"=>false, "merge_user_id"=>nil, "merge_commit_sha"=>nil, "approvals_before_merge"=>nil, "rebase_commit_sha"=>nil, "in_progress_merge_commit_sha"=>nil, "lock_version"=>0, "time_estimate"=>0, "squash"=>false, "last_edited_at"=>nil, "last_edited_by_id"=>nil, "head_pipeline_id"=>640263, "merge_jid"=>nil, "discussion_locked"=>nil, "latest_merge_request_diff_id"=>495888, "allow_maintainer_to_push"=>nil, "state_id"=>2}
irb(main):009:0> mr.state
=> "closed"
irb(main):010:0> mr.has_ci?
=> true
irb(main):011:0> mr.broken?
=> false
irb(main):012:0> mr.branch_missing?
=> false
irb(main):013:0> mr.head_pipeline
=> #<Ci::Pipeline id: 640263, ref: "something-bugfix/other-words-here", sha: "461d1621094d547866a0985ef8a2b95d96ae3247", before_sha: nil, created_at: "2019-11-04 18:52:43", updated_at: "2019-11-04 18:57:49", tag: false, yaml_errors: nil, committed_at: nil, project_id: 21708, status: "success", started_at: "2019-11-04 18:52:58", finished_at: "2019-11-04 18:57:49", duration: 0, user_id: 7283, lock_version: 3, auto_canceled_by_id: nil, pipeline_schedule_id: nil, source: "external", protected: false, config_source: nil, failure_reason: nil, iid: 691, merge_request_id: nil, source_sha: nil, target_sha: nil>
irb(main):014:0> mr.actual_head_pipeline
=> nil
irb(main):015:0> mr.base_pipeline
=> nil
irb(main):016:0> mr.head_pipeline.attributes
=> {"id"=>640263, "ref"=>"something-bugfix/other-words-here", "sha"=>"461d1621094d547866a0985ef8a2b95d96ae3247", "before_sha"=>nil, "created_at"=>Mon, 04 Nov 2019 18:52:43 UTC +00:00, "updated_at"=>Mon, 04 Nov 2019 18:57:49 UTC +00:00, "tag"=>false, "yaml_errors"=>nil, "committed_at"=>nil, "project_id"=>21708, "status"=>"success", "started_at"=>Mon, 04 Nov 2019 18:52:58 UTC +00:00, "finished_at"=>Mon, 04 Nov 2019 18:57:49 UTC +00:00, "duration"=>0, "user_id"=>7283, "lock_version"=>3, "auto_canceled_by_id"=>nil, "pipeline_schedule_id"=>nil, "source"=>"external", "protected"=>false, "config_source"=>nil, "failure_reason"=>nil, "iid"=>691, "merge_request_id"=>nil, "source_sha"=>nil, "target_sha"=>nil}
irb(main):017:0> mr.reload_head_pipeline
=> #<Ci::Pipeline id: 640263, ref: "something-bugfix/other-words-here", sha: "461d1621094d547866a0985ef8a2b95d96ae3247", before_sha: nil, created_at: "2019-11-04 18:52:43", updated_at: "2019-11-04 18:57:49", tag: false, yaml_errors: nil, committed_at: nil, project_id: 21708, status: "success", started_at: "2019-11-04 18:52:58", finished_at: "2019-11-04 18:57:49", duration: 0, user_id: 7283, lock_version: 3, auto_canceled_by_id: nil, pipeline_schedule_id: nil, source: "external", protected: false, config_source: nil, failure_reason: nil, iid: 691, merge_request_id: nil, source_sha: nil, target_sha: nil>
irb(main):018:0> mr.update_head_pipeline
=> true
irb(main):019:0> mr.head_pipeline.attributes
=> {"id"=>640234, "ref"=>"something-bugfix/other-words-here", "sha"=>"c7feb6c61525ecb4c9a010ccd095e57367d004f6", "before_sha"=>nil, "created_at"=>Mon, 04 Nov 2019 18:38:31 UTC +00:00, "updated_at"=>Mon, 04 Nov 2019 18:43:47 UTC +00:00, "tag"=>false, "yaml_errors"=>nil, "committed_at"=>nil, "project_id"=>21708, "status"=>"success", "started_at"=>Mon, 04 Nov 2019 18:38:48 UTC +00:00, "finished_at"=>Mon, 04 Nov 2019 18:43:47 UTC +00:00, "duration"=>0, "user_id"=>7283, "lock_version"=>3, "auto_canceled_by_id"=>nil, "pipeline_schedule_id"=>nil, "source"=>"external", "protected"=>false, "config_source"=>nil, "failure_reason"=>nil, "iid"=>690, "merge_request_id"=>nil, "source_sha"=>nil, "target_sha"=>nil}
irb(main):024:0> Ci::Pipeline.find 640263
=> #<Ci::Pipeline id: 640263, ref: "something-bugfix/other-words-here", sha: "461d1621094d547866a0985ef8a2b95d96ae3247", before_sha: nil, created_at: "2019-11-04 18:52:43", updated_at: "2019-11-04 18:57:49", tag: false, yaml_errors: nil, committed_at: nil, project_id: 21708, status: "success", started_at: "2019-11-04 18:52:58", finished_at: "2019-11-04 18:57:49", duration: 0, user_id: 7283, lock_version: 3, auto_canceled_by_id: nil, pipeline_schedule_id: nil, source: "external", protected: false, config_source: nil, failure_reason: nil, iid: 691, merge_request_id: nil, source_sha: nil, target_sha: nil>
irb(main):025:0> Ci::Pipeline.find 640234
=> #<Ci::Pipeline id: 640234, ref: "something-bugfix/other-words-here", sha: "c7feb6c61525ecb4c9a010ccd095e57367d004f6", before_sha: nil, created_at: "2019-11-04 18:38:31", updated_at: "2019-11-04 18:43:47", tag: false, yaml_errors: nil, committed_at: nil, project_id: 21708, status: "success", started_at: "2019-11-04 18:38:48", finished_at: "2019-11-04 18:43:47", duration: 0, user_id: 7283, lock_version: 3, auto_canceled_by_id: nil, pipeline_schedule_id: nil, source: "external", protected: false, config_source: nil, failure_reason: nil, iid: 690, merge_request_id: nil, source_sha: nil, target_sha: nil>

Output of checks

This bug happens on GitLab.com

Also happens on self-managed instances

Possible fixes

Not sure, but here's the method that's returning nil when we don't expect it to.

models/merge_request.rb#L281-283

Workaround

If you experience this same issue, then try this in the rails console (sudo gitlab-rails console):

project = Project.find_by_full_path('my-group/example-project')
# you can find the iid of the MR by looking at the number at the end of the URL when viewing the MR
mr = MergeRequest.find_by(project: project, iid: 154)
mr.update_head_pipeline