One in two requests on /repository/project/pipelines/pipelines.json getting HTTP 429 after upgrading Gitlab EE to latest version(s)

Summary

One in two requests to /repository/project/pipelines/pipelines.json is returning a HTTP 429. This happened after upgrading Dockerized Gitlab from gitlab/gitlab-ee:11.4.7-ee.0 to gitlab/gitlab-ee:11.9.9-ee.0. I tried upgrading to the latest images as of yet namely gitlab/gitlab-ee:11.9.11-ee.0 and gitlab/gitlab-ee:11.10.4-ee.0 and the same issue persists.

Every repository with pipeline history suffers the issue.

The logs are not showing what is causing the 429 so I don't know what the cause is.

As a side effect, the fact that a 429 happens every 1 in 2 requests, makes the pipeline stage buttons completely unusable on that page. This is because the pipeline button calls pipelines.json?scope=all&page=1 first, which throws a HTTP 429 causing the entire page to show that error message, followed by a call to stage.json?scope=myscope which returns a HTTP 200, followed again by calling pipelines.json?scope=all&page=1 again which succeeds with a HTTP 200 and re-renders the pipelines page a few seconds later.

Steps to reproduce

Simply go to a repository at gitlab.namespace.com/repository/project/pipelines. Every one in two visits, the message "There was an error fetching the pipelines. Try again in a few moments or contact your support team." appears.

Example Project

None. This does not happen on Gitlab.com

What is the current bug behavior?

Looking at the developer tools, the request to /repository/project/pipelines/pipelines.json?scope=al&page=1 is returning a 429.

What is the expected correct behavior?

A working pipelines page with no HTTP 429 and big error message.

Relevant logs and/or screenshots

Request header:

Host: gitlab.namespace.com
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0
Accept: application/json, text/plain, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://gitlab.namespace.com/namespace/repository/pipelines
X-CSRF-Token: xxxxxxxxxxxxxxxx
X-Requested-With: XMLHttpRequest
DNT: 1
Connection: keep-alive
Cookie: sidebar_collapsed=false; diff_view=parallel; _gitlab_session=xxxxxxxxxxxxxxxx; event_filter=all; remember_user_token=xxxxxxxxxxxxxxxx
If-None-Match: W/"66b557092b067af4e440e8eee1979337"
TE: Trailers

Response header:

HTTP/2.0 429 No Reason Phrase
cache-control: no-cache
date: Fri, 03 May 2019 07:11:12 GMT
etag: W/"66b557092b067af4e440e8eee1979337"
server: nginx
x-gitlab-from-cache: true
x-request-id: lCy4NKwtTE7
x-runtime: 0.001312
content-length: 0
X-Firefox-Spdy: h2

Typical dockerized Gitlab logs (from gitlab-ctl tail) for a request to /repository/project/pipelines:

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/namespace/repository/pipelines" for 127.0.0.1 at 2019-05-03 07:11:11 +0000
Processing by Projects::PipelinesController#index as HTML
  Parameters: {"namespace_id"=>"namespace", "project_id"=>"repository"}
Completed 200 OK in 106ms (Views: 77.3ms | ActiveRecord: 8.6ms | Elasticsearch: 0.0ms)

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/namespace/repository/pipelines","format":"html","controller":"Projects::PipelinesController","action":"index","status":200,"duration":109.46,"view":77.26,"db":8.64,"time":"2019-05-03T07:11:11.380Z","params":[{"key":"namespace_id","value":"namespace"},{"key":"project_id","value":"repository"}],"remote_ip":"127.0.0.1","user_id":3,"username":"leojonathanoh","ua":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0","queue_duration":null,"correlation_id":"939uCmXcX77"}

==> /var/log/gitlab/nginx/gitlab_access.log <==
192.168.0.1 - - [03/May/2019:07:11:11 +0000] "GET /namespace/repository/pipelines HTTP/1.1" 200 9111 "https://gitlab.namespace.com/namespace/repository/pipelines" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0"

==> /var/log/gitlab/gitlab-workhorse/current <==
2019-05-03_07:11:11.49151 gitlab.namespace.com 127.0.0.1:0 - - [2019/05/03:07:11:11 +0000] "GET /namespace/repository/pipelines HTTP/1.1" 200 34948 "https://gitlab.namespace.com/namespace/repository/pipelines" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.123

==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/namespace/repository/pipelines.json?scope=all&page=1" for 127.0.0.1 at 2019-05-03 07:11:12 +0000
Started GET "/uploads/-/system/group/avatar/5/pic.png?width=15" for 127.0.0.1 at 2019-05-03 07:11:12 +0000
Processing by UploadsController#show as HTML
  Parameters: {"width"=>"15", "model"=>"group", "mounted_as"=>"avatar", "id"=>"5", "filename"=>"pic.png"}
Sent file /opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/system/group/avatar/5/pic.png (0.1ms)
Completed 200 OK in 13ms (ActiveRecord: 2.1ms | Elasticsearch: 0.0ms)

==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/uploads/-/system/group/avatar/5/pic.png","format":"html","controller":"UploadsController","action":"show","status":200,"duration":13.39,"view":0.0,"db":2.08,"time":"2019-05-03T07:11:12.582Z","params":[{"key":"width","value":"15"},{"key":"model","value":"group"},{"key":"mounted_as","value":"avatar"},{"key":"id","value":"5"},{"key":"filename","value":"pic.png"}],"remote_ip":"127.0.0.1","user_id":3,"username":"leojonathanoh","ua":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0","queue_duration":null,"correlation_id":"ZVQPQg8eQM1"}

==> /var/log/gitlab/nginx/gitlab_access.log <==
192.168.0.1 - - [03/May/2019:07:11:12 +0000] "GET /namespace/repository/pipelines.json?scope=all&page=1 HTTP/1.1" 429 0 "https://gitlab.namespace.com/namespace/repository/pipelines" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0"
192.168.0.1 - - [03/May/2019:07:11:12 +0000] "GET /uploads/-/system/group/avatar/5/pic.png?width=15 HTTP/1.1" 200 89622 "https://gitlab.namespace.com/namespace/repository/pipelines" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0"

==> /var/log/gitlab/gitlab-workhorse/current <==
2019-05-03_07:11:12.20368 gitlab.namespace.com 127.0.0.1:0 - - [2019/05/03:07:11:12 +0000] "GET /namespace/repository/pipelines.json?scope=all&page=1 HTTP/1.1" 429 0 "https://gitlab.namespace.com/namespace/repository/pipelines" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.002
2019-05-03_07:11:12.59634 time="2019-05-03T07:11:12Z" level=info msg="Send file" correlation_id=ZVQPQg8eQM1 file=/opt/gitlab/embedded/service/gitlab-rails/public/uploads/-/system/group/avatar/5/pic.png method=GET uri="/uploads/-/system/group/avatar/5/pic.png?width=15"
2019-05-03_07:11:12.59659 gitlab.namespace.com 127.0.0.1:0 - - [2019/05/03:07:11:12 +0000] "GET /uploads/-/system/group/avatar/5/pic.png?width=15 HTTP/1.1" 200 89622 "https://gitlab.namespace.com/namespace/repository/pipelines" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:66.0) Gecko/20100101 Firefox/66.0" 0.020

Output of checks

Results of GitLab environment info

System information
System:
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.5.3p105
Gem Version:    2.7.6
Bundler Version:1.16.6
Rake Version:   12.3.2
Redis Version:  3.2.12
Git Version:    2.18.1
Sidekiq Version:5.2.5
Go Version:     unknown

GitLab information
Version:        11.9.9-ee
Revision:       0ee466e
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
DB Version:     9.6.11
URL:            https://gitlab.namespace.com
HTTP Clone URL: https://gitlab.namespace.com/some-group/some-project.git
SSH Clone URL:  git@gitlab.namespace.com:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: 

GitLab Shell
Version:        8.7.1
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 8.7.1 ? ... OK (8.7.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK

Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ... 
3/1 ... yes
4/3 ... yes
4/4 ... yes
4/5 ... yes
4/6 ... yes
3/7 ... yes
3/8 ... yes
3/9 ... yes
3/10 ... yes
3/11 ... yes
3/12 ... yes
3/28 ... yes
4/30 ... yes
4/31 ... yes
5/35 ... yes
5/36 ... yes
5/37 ... yes
5/38 ... yes
5/39 ... yes
5/40 ... yes
5/41 ... yes
5/42 ... yes
3/43 ... yes
4/44 ... yes
5/48 ... yes
3/49 ... yes
5/50 ... yes
5/51 ... yes
5/52 ... yes
5/53 ... yes
5/54 ... yes
4/56 ... yes
4/57 ... yes
4/58 ... yes
4/59 ... yes
5/60 ... yes
3/61 ... yes
4/62 ... yes
5/63 ... yes
3/66 ... yes
3/69 ... yes
3/70 ... yes
3/71 ... yes
3/77 ... yes
3/78 ... yes
5/79 ... yes
5/80 ... yes
3/81 ... yes
3/82 ... yes
5/83 ... yes
4/84 ... yes
5/85 ... yes
5/88 ... yes
5/90 ... yes
5/91 ... yes
5/96 ... yes
3/102 ... yes
5/104 ... yes
5/107 ... yes
5/108 ... yes
5/109 ... yes
5/110 ... yes
5/114 ... yes
3/116 ... yes
3/117 ... yes
3/118 ... yes
5/120 ... yes
5/121 ... yes
3/123 ... yes
5/124 ... yes
5/134 ... yes
5/135 ... yes
3/137 ... yes
4/141 ... yes
5/142 ... yes
5/143 ... yes
4/144 ... yes
3/145 ... yes
3/146 ... yes
4/147 ... yes
5/148 ... yes
3/149 ... yes
5/150 ... yes
3/151 ... yes
5/152 ... yes
5/154 ... yes
3/155 ... yes
5/156 ... yes
3/158 ... yes
3/159 ... yes
5/162 ... yes
5/163 ... yes
3/164 ... yes
3/165 ... yes
3/166 ... yes
3/167 ... yes
3/168 ... yes
3/169 ... yes
5/170 ... yes
5/171 ... yes
5/172 ... yes
5/173 ... yes
3/174 ... yes
4/175 ... yes
3/176 ... yes
3/177 ... yes
3/178 ... yes
4/180 ... yes
3/181 ... yes
5/182 ... yes
3/183 ... yes
5/184 ... yes
5/185 ... yes
5/186 ... yes
5/187 ... yes
4/188 ... yes
5/189 ... yes
5/191 ... yes
3/192 ... yes
5/193 ... yes
5/194 ... yes
5/195 ... yes
5/196 ... yes
5/197 ... yes
3/198 ... yes
3/199 ... yes
5/200 ... yes
4/201 ... yes
5/202 ... yes
5/203 ... yes
5/204 ... yes
5/205 ... yes
3/206 ... yes
5/207 ... yes
5/208 ... yes
5/209 ... yes
5/210 ... yes
5/211 ... yes
5/212 ... yes
5/213 ... yes
5/214 ... yes
4/215 ... yes
5/216 ... yes
3/217 ... yes
3/218 ... yes
3/219 ... yes
3/220 ... yes
3/221 ... yes
3/222 ... yes
3/223 ... yes
3/224 ... yes
3/225 ... yes
3/226 ... yes
3/227 ... yes
3/228 ... yes
3/229 ... yes
3/230 ... yes
3/231 ... yes
3/232 ... yes
5/233 ... yes
3/234 ... yes
3/235 ... yes
4/236 ... yes
5/237 ... yes
5/238 ... yes
5/239 ... yes
3/240 ... yes
4/241 ... yes
4/242 ... yes
4/243 ... yes
4/244 ... yes
5/245 ... yes
3/246 ... yes
5/249 ... yes
5/251 ... yes
5/252 ... yes
5/253 ... yes
5/254 ... yes
4/255 ... yes
4/256 ... yes
5/257 ... yes
4/258 ... yes
5/259 ... yes
3/260 ... yes
5/261 ... yes
4/262 ... yes
5/263 ... yes
3/264 ... yes
4/265 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.5.3)
Git version >= 2.18.0 ? ... yes (2.18.1)
Git user has default SSH configuration? ... yes
Active users: ... 4
Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

Edited May 03, 2019 by leo