Could not retrieve the pipeline status (and Pipelines tab stops updating)
Summary
The MR widget shows Could not retrieve the pipeline status
instead of pipeline status. Additionally the pipeline tab stops updating.
Further investigation by @weimeng (see detail in comments) suggests that a 10,000 limit within the code base is causing this.
def all_commits MergeRequestDiffCommit .where(merge_request_diff: merge_request_diffs.recent) .limit(10_000) end
This issue is almost identical to #36781 (closed) except that in this case, the tab stops updating.
It's also similar to #27111 (closed); and I've asked for some more details from a recent update because another customer reports a possible third variant in the comments:
The pipeline triggered by MR merge says "No related merge requests found.".
As I've reproduced it, I don't see that.
This issue also has another property: the MR is huge; as reproduced, 1800+ commits, over 1000 changes, and 29 pipelines. This is because the customer who reported it (internal ticket linked) indicated that they were seeing the issue on MRs with these sorts of parameters.
#24723 is also similar; pipeline tab stops updating in that issue; there's lots of commits, like this issue. However, the pipeline status widget is still working on that other issue, which it isn't here.
Steps to reproduce
- GitLab 12.6.3
- New repo; with some CI:
stages:
- one
_one:
stage: one
script:
- /bin/true
Create a branch
git checkout -b branch1 origin/master
mkdir obj ; touch obj/foo ; git add obj
git commit -m 'init'
git push -u origin HEAD
Create an MR.
We're now going to load up the MR by creating a huge number of files and commits.
for i in {10..27} ; do
for j in {00..99}; do
k="${i}${j}" ; mkdir obj/$k ; date > obj/$k/001 ; git add obj/$k ; git commit -m "add obj/$k"
done
echo $k ; sleep 5 ; git push -u origin HEAD
done
NB: the sleep is timed to ensure the pipelines run to success. This may or may not be important, but I found that if I looped too fast, the pipelines got cancelled.
Status of MR:
- commits: 1802
- pipelines: 20
- changes: 1000+
Creating other activity in the project seems to be key to this.
Update: this has been repoduced with no further steps - on gitlab.com
git checkout -b branch2
for i in {2800..2809} ; do mkdir obj/$i ; date > obj/$i/001 ; git add obj/$i ; git commit -m "add obj/$i" ; done
git push -u origin HEAD
Create a MR for that one as well.
We then go back to the original branch and start pushing commits and creating pipelines.
while this loop is running, we're checking the status of the pipelines and we're reloading the MR. I don't think this is important for reproducing the bug.
In my experience, the bug reproduces in full by the time the loop has run 20 times.
You may get Could not retrieve the pipeline status.
intermittently, but it goes away on a refresh. Once reproduced, you will not be able reload and clear this state, and the pipelines tab is definitely not updating any more.
git checkout branch1
for i in {2810..2840} ; do
mkdir obj/$i ; date > obj/$i/001 ; git add obj/$i ; git commit -m "add obj/$i"; git push -u origin HEAD
sleep 30
done
Example Project
This has been reproduced on gitlab.com using the reproduction script in this issue.
What is the current bug behavior?
MR gets stuck in Could not retrieve the pipeline status.
and pipelines tab stops updating.
What is the expected correct behavior?
MR pipeline status and history is updated.
Relevant logs and/or screenshots
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
[root@bprescott-gitlabtest-0 ~]# sudo gitlab-rake gitlab:env:info
System information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.3p62 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 3.2.12 Git Version: 2.24.1 Sidekiq Version:5.2.7 Go Version: unknown
GitLab information Version: 12.6.3-ee Revision: 461e0ee8310 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.9 URL: https://bprescott-gitlabtest-0.do.gitlap.com HTTP Clone URL: https://bprescott-gitlabtest-0.do.gitlap.com/some-group/some-project.git SSH Clone URL: git@bprescott-gitlabtest-0.do.gitlap.com:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 10.3.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
[root@bprescott-gitlabtest-0 ~]# sudo gitlab-rake gitlab:env:infoSystem information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.3p62 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 3.2.12 Git Version: 2.24.1 Sidekiq Version:5.2.7 Go Version: unknown
GitLab information Version: 12.6.3-ee Revision: 461e0ee8310 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.9 URL: https://bprescott-gitlabtest-0.do.gitlap.com HTTP Clone URL: https://bprescott-gitlabtest-0.do.gitlap.com/some-group/some-project.git SSH Clone URL: git@bprescott-gitlabtest-0.do.gitlap.com:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 10.3.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git [root@bprescott-gitlabtest-0 ~]# sudo gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 10.3.0 ? ... OK (10.3.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 2/1 ... yes 2/2 ... yes 2/3 ... yes 2/4 ... yes 2/6 ... yes 2/7 ... yes 4/8 ... yes 2/9 ... yes 2/10 ... yes 2/11 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.3) Git version >= 2.22.0 ? ... yes (2.24.1) Git user has default SSH configuration? ... yes Active users: ... 1 Is authorized keys file accessible? ... yes Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
Investigation suggests that a limit in the code is involved with causing this issue.