Performance problem - gitaly timeouts
Summary
After upgrade gitlab-ce to latest 12.7.5 version, we have problem with browsing project repository, cloning, merge request (all work with repository and git) in all repositories. I think that this is problem with gitaly. Into logs is many errors with timeout ...
Steps to reproduce
exactly after uprgade to this version
What is the current bug behavior?
Very slow response while browsing repository and many times it end with 503 Unable to clone repository, push to repository, create and merge merge requests, ...
What is the expected correct behavior?
no error with gitaly or other subsystem
Relevant logs and/or screenshots
Expand for output related to gitaly.log
{"correlation_id":"myFSTM6G5B3","error":"rpc error: code = Canceled desc = rpc error: code = Canceled desc = context canceled","grpc.code":"Canceled","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"GetLFSPointers","grpc.request.deadline":"2020-02-05T15:23:30Z","grpc.request.fullMethod":"/gitaly.BlobService/GetLFSPointers","grpc.request.glProjectPath":"***/***","grpc.request.glRepository":"project-334","grpc.request.repoPath":"***/***.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"***","grpc.service":"gitaly.BlobService","grpc.start_time":"2020-02-05T15:22:33Z","grpc.time_ms":56821.24,"level":"info","msg":"finished streaming call with code Canceled","peer.address":"@","pid":29046,"span.kind":"server","system":"grpc","time":"2020-02-05T15:23:30Z"} {"correlation_id":"j54ulileDP3","error":"rpc error: code = Canceled desc = rpc error: code = Canceled desc = context canceled","grpc.code":"Canceled","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"GetLFSPointers","grpc.request.deadline":"2020-02-05T15:23:45Z","grpc.request.fullMethod":"/gitaly.BlobService/GetLFSPointers","grpc.request.glProjectPath":"***/***","grpc.request.glRepository":"project-451","grpc.request.repoPath":"***/***.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"***","grpc.service":"gitaly.BlobService","grpc.start_time":"2020-02-05T15:22:48Z","grpc.time_ms":56198.066,"level":"info","msg":"finished streaming call with code Canceled","peer.address":"@","pid":29046,"span.kind":"server","system":"grpc","time":"2020-02-05T15:23:45Z"} {"correlation_id":"QlejZiiJn98","error":"rpc error: code = Canceled desc = rpc error: code = Canceled desc = context canceled","grpc.code":"Canceled","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"ListConflictFiles","grpc.request.deadline":"2020-02-05T15:24:04Z","grpc.request.fullMethod":"/gitaly.ConflictsService/ListConflictFiles","grpc.request.glProjectPath":"***/***","grpc.request.glRepository":"project-733","grpc.request.repoPath":"***/***.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"***","grpc.service":"gitaly.ConflictsService","grpc.start_time":"2020-02-05T15:23:07Z","grpc.time_ms":56102.77,"level":"info","msg":"finished streaming call with code Canceled","peer.address":"@","pid":29046,"span.kind":"server","system":"grpc","time":"2020-02-05T15:24:04Z"} {"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","level":"warning","msg":"health check failed","time":"2020-02-05T15:24:26Z","worker.name":"gitaly-ruby.1"} {"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","level":"warning","msg":"health check failed","time":"2020-02-05T15:24:42Z","worker.name":"gitaly-ruby.1"} {"correlation_id":"mNUkeNK11m8","error":"rpc error: code = Canceled desc = rpc error: code = Canceled desc = context canceled","grpc.code":"Canceled","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"GetLFSPointers","grpc.request.deadline":"2020-02-05T15:24:56Z","grpc.request.fullMethod":"/gitaly.BlobService/GetLFSPointers","grpc.request.glProjectPath":"***/***","grpc.request.glRepository":"project-774","grpc.request.repoPath":"***/***.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"***","grpc.service":"gitaly.BlobService","grpc.start_time":"2020-02-05T15:23:59Z","grpc.time_ms":56083.4,"level":"info","msg":"finished streaming call with code Canceled","peer.address":"@","pid":29046,"span.kind":"server","system":"grpc","time":"2020-02-05T15:24:56Z"} {"correlation_id":"n0qUQjLt1U4","error":"rpc error: code = Canceled desc = rpc error: code = Canceled desc = context canceled","grpc.code":"Canceled","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"GetLFSPointers","grpc.request.deadline":"2020-02-05T15:24:58Z","grpc.request.fullMethod":"/gitaly.BlobService/GetLFSPointers","grpc.request.glProjectPath":"***/***","grpc.request.glRepository":"project-602","grpc.request.repoPath":"***/***.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"***","grpc.service":"gitaly.BlobService","grpc.start_time":"2020-02-05T15:24:01Z","grpc.time_ms":56569.973,"level":"info","msg":"finished streaming call with code Canceled","peer.address":"@","pid":29046,"span.kind":"server","system":"grpc","time":"2020-02-05T15:24:58Z"} {"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","level":"warning","msg":"health check failed","time":"2020-02-05T15:24:58Z","worker.name":"gitaly-ruby.1"} {"correlation_id":"EOoXhC1yze9","error":"rpc error: code = Canceled desc = rpc error: code = Canceled desc = context canceled","grpc.code":"Canceled","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"GetLFSPointers","grpc.request.deadline":"2020-02-05T15:25:02Z","grpc.request.fullMethod":"/gitaly.BlobService/GetLFSPointers","grpc.request.glProjectPath":"***/***","grpc.request.glRepository":"project-485","grpc.request.repoPath":"***/***.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"***","grpc.service":"gitaly.BlobService","grpc.start_time":"2020-02-05T15:24:05Z","grpc.time_ms":56015.008,"level":"info","msg":"finished streaming call with code Canceled","peer.address":"@","pid":29046,"span.kind":"server","system":"grpc","time":"2020-02-05T15:25:02Z"} {"correlation_id":"9x5vkxtbYea","error":"rpc error: code = Canceled desc = rpc error: code = Canceled desc = context canceled","grpc.code":"Canceled","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"ListConflictFiles","grpc.request.deadline":"2020-02-05T15:25:04Z","grpc.request.fullMethod":"/gitaly.ConflictsService/ListConflictFiles","grpc.request.glProjectPath":"***/***","grpc.request.glRepository":"project-733","grpc.request.repoPath":"***/***.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"***","grpc.service":"gitaly.ConflictsService","grpc.start_time":"2020-02-05T15:24:07Z","grpc.time_ms":56197.074,"level":"info","msg":"finished streaming call with code Canceled","peer.address":"@","pid":29046,"span.kind":"server","system":"grpc","time":"2020-02-05T15:25:04Z"} {"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","level":"warning","msg":"health check failed","time":"2020-02-05T15:25:14Z","worker.name":"gitaly-ruby.1"} {"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","level":"warning","msg":"health check failed","time":"2020-02-05T15:25:30Z","worker.name":"gitaly-ruby.1"} {"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","level":"warning","msg":"health check failed","time":"2020-02-05T15:25:46Z","worker.name":"gitaly-ruby.1"} {"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","level":"warning","msg":"health check failed","time":"2020-02-05T15:26:02Z","worker.name":"gitaly-ruby.1"} {"correlation_id":"gkjwt2IRvI4","error":"rpc error: code = Canceled desc = rpc error: code = Canceled desc = context canceled","grpc.code":"Canceled","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"ListConflictFiles","grpc.request.deadline":"2020-02-05T15:26:05Z","grpc.request.fullMethod":"/gitaly.ConflictsService/ListConflictFiles","grpc.request.glProjectPath":"***/***","grpc.request.glRepository":"project-733","grpc.request.repoPath":"***/***.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"***","grpc.service":"gitaly.ConflictsService","grpc.start_time":"2020-02-05T15:25:08Z","grpc.time_ms":56947.88,"level":"info","msg":"finished streaming call with code Canceled","peer.address":"@","pid":29046,"span.kind":"server","system":"grpc","time":"2020-02-05T15:26:05Z"} {"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","level":"warning","msg":"health check failed","time":"2020-02-05T15:26:09Z","worker.name":"gitaly-ruby.0"} {"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","level":"warning","msg":"health check failed","time":"2020-02-05T15:26:25Z","worker.name":"gitaly-ruby.0"} {"error":null,"level":"warning","msg":"exited","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","29046","/var/opt/gitlab/gitaly/internal_sockets/ruby.1"],"supervisor.name":"gitaly-ruby.1","time":"2020-02-05T15:26:30Z"} {"error":"rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/internal_sockets/ruby.1: connect: no such file or directory\"","level":"warning","msg":"health check failed","time":"2020-02-05T15:26:32Z","worker.name":"gitaly-ruby.1"} {"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","level":"warning","msg":"health check failed","time":"2020-02-05T15:26:41Z","worker.name":"gitaly-ruby.0"} {"error":"rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/internal_sockets/ruby.1: connect: no such file or directory\"","level":"warning","msg":"health check failed","time":"2020-02-05T15:26:47Z","worker.name":"gitaly-ruby.1"} {"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","level":"warning","msg":"health check failed","time":"2020-02-05T15:26:57Z","worker.name":"gitaly-ruby.0"} {"error":"rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/internal_sockets/ruby.1: connect: no such file or directory\"","level":"warning","msg":"health check failed","time":"2020-02-05T15:27:02Z","worker.name":"gitaly-ruby.1"} {"correlation_id":"EiWfKOOSXJ8","error":"rpc error: code = Canceled desc = rpc error: code = Canceled desc = context canceled","grpc.code":"Canceled","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"ListConflictFiles","grpc.request.deadline":"2020-02-05T15:27:04Z","grpc.request.fullMethod":"/gitaly.ConflictsService/ListConflictFiles","grpc.request.glProjectPath":"***/***","grpc.request.glRepository":"project-733","grpc.request.repoPath":"***/***.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"***","grpc.service":"gitaly.ConflictsService","grpc.start_time":"2020-02-05T15:26:07Z","grpc.time_ms":56203.23,"level":"info","msg":"finished streaming call with code Canceled","peer.address":"@","pid":29046,"span.kind":"server","system":"grpc","time":"2020-02-05T15:27:04Z"} {"correlation_id":"NKl4FsPzyW3","error":"rpc error: code = Canceled desc = rpc error: code = Canceled desc = context canceled","grpc.code":"Canceled","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"GetLFSPointers","grpc.request.deadline":"2020-02-05T15:27:11Z","grpc.request.fullMethod":"/gitaly.BlobService/GetLFSPointers","grpc.request.glProjectPath":"***/***","grpc.request.glRepository":"project-645","grpc.request.repoPath":"***/***.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"***","grpc.service":"gitaly.BlobService","grpc.start_time":"2020-02-05T15:26:14Z","grpc.time_ms":56623.426,"level":"info","msg":"finished streaming call with code Canceled","peer.address":"@","pid":29046,"span.kind":"server","system":"grpc","time":"2020-02-05T15:27:11Z"} {"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","level":"warning","msg":"health check failed","time":"2020-02-05T15:27:13Z","worker.name":"gitaly-ruby.0"} {"error":"rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/internal_sockets/ruby.1: connect: no such file or directory\"","level":"warning","msg":"health check failed","time":"2020-02-05T15:27:17Z","worker.name":"gitaly-ruby.1"} {"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","level":"warning","msg":"health check failed","time":"2020-02-05T15:27:29Z","worker.name":"gitaly-ruby.0"} {"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","level":"warning","msg":"health check failed","time":"2020-02-05T15:27:45Z","worker.name":"gitaly-ruby.0"} {"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","level":"warning","msg":"health check failed","time":"2020-02-05T15:28:01Z","worker.name":"gitaly-ruby.0"} {"correlation_id":"yvOAZR6Aqt7","error":"rpc error: code = Canceled desc = rpc error: code = Canceled desc = context canceled","grpc.code":"Canceled","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"ListConflictFiles","grpc.request.deadline":"2020-02-05T15:28:04Z","grpc.request.fullMethod":"/gitaly.ConflictsService/ListConflictFiles","grpc.request.glProjectPath":"***/***","grpc.request.glRepository":"project-733","grpc.request.repoPath":"***/***.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"***","grpc.service":"gitaly.ConflictsService","grpc.start_time":"2020-02-05T15:27:07Z","grpc.time_ms":56221.96,"level":"info","msg":"finished streaming call with code Canceled","peer.address":"@","pid":29046,"span.kind":"server","system":"grpc","time":"2020-02-05T15:28:04Z"} {"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded","level":"warning","msg":"health check failed","time":"2020-02-05T15:28:17Z","worker.name":"gitaly-ruby.0"}
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Gitlab is installed on CentOS7, installed over yum. 8cores 16G RAM
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.6.5p114 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.7 Git Version: 2.24.1 Sidekiq Version:5.2.7 Go Version: go1.13.3 linux/amd64
GitLab information Version: 12.7.5 Revision: b0fe37ef58e Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 9.6.16 URL: https://gitlab.**** HTTP Clone URL: https://gitlab./some-group/some-project.git SSH Clone URL: git@gitlab.:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: google_oauth2, facebook, bitbucket, github
GitLab Shell Version: 11.0.0 Repository storage paths: default: /opt/gitlab-git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 11.0.0 ? ... OK (11.0.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Checking Reply by email ...
IMAP server credentials are correct? ... yes Init.d configured correctly? ... skipped MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 2/1 ... yes 6/2 ... yes 2/3 ... yes 2/5 ... yes 2/6 ... yes 5/8 ... yes 3/10 ... yes 17/11 ... yes 22/12 ... yes 9/13 ... yes 23/17 ... yes 23/20 ... yes 23/22 ... yes 23/23 ... yes 23/24 ... yes 23/25 ... yes 23/26 ... yes 23/27 ... yes 23/28 ... yes 23/29 ... yes 5/30 ... yes 23/31 ... yes 4/32 ... yes 23/33 ... yes 15/35 ... yes 15/36 ... yes 5/37 ... yes 14/40 ... yes 4/41 ... yes 3/42 ... yes 20/43 ... yes 4/44 ... yes 7/45 ... yes 9/46 ... yes 6/47 ... yes 23/49 ... yes 20/50 ... yes 27/51 ... yes 29/52 ... yes 3/53 ... yes 20/54 ... yes 14/55 ... yes 30/59 ... yes 14/60 ... yes 2/62 ... yes 32/63 ... yes 32/64 ... yes 32/67 ... yes 2/68 ... yes 36/69 ... yes 36/70 ... yes 36/71 ... yes 32/73 ... yes 4/74 ... yes 32/75 ... yes 40/76 ... yes 4/77 ... yes 129/78 ... yes 32/79 ... yes 2/80 ... yes 2/81 ... yes 41/82 ... yes 18/85 ... yes 82/86 ... yes 32/87 ... yes 32/88 ... yes 96/89 ... yes 32/90 ... yes 2/91 ... yes 2/92 ... yes 2/93 ... yes 41/94 ... yes 18/95 ... yes 4/96 ... yes 45/99 ... yes 45/100 ... yes 45/101 ... yes 36/102 ... yes 45/103 ... yes 46/104 ... yes 46/106 ... yes 18/107 ... yes 46/108 ... yes 46/109 ... yes 26/112 ... yes 48/114 ... yes 48/116 ... yes 48/117 ... yes 2/118 ... yes 3/120 ... yes 14/123 ... yes 41/124 ... yes 14/125 ... yes 36/126 ... yes 32/127 ... yes 32/128 ... yes 50/129 ... yes 2/130 ... yes 14/131 ... yes 3/132 ... yes 8/133 ... yes 20/134 ... yes 20/135 ... yes 14/136 ... yes 14/137 ... yes 54/138 ... yes 41/139 ... yes 79/140 ... yes 56/142 ... yes 57/143 ... yes 56/145 ... yes 50/148 ... yes 24/149 ... yes 2/150 ... yes 2/151 ... yes 14/152 ... yes 9/153 ... yes 48/154 ... yes 4/156 ... yes 48/158 ... yes 9/161 ... yes 14/164 ... yes 9/166 ... yes 37/167 ... yes 24/170 ... yes 14/171 ... yes 326/173 ... yes 326/174 ... yes 14/175 ... yes 49/178 ... yes 49/179 ... yes 49/180 ... yes 49/181 ... yes 49/182 ... yes 49/183 ... yes 49/184 ... yes 49/185 ... yes 63/186 ... yes 14/187 ... yes 49/188 ... yes 46/189 ... yes 3/190 ... yes 64/191 ... yes 48/192 ... yes 46/193 ... yes 321/194 ... yes 46/195 ... yes 50/196 ... yes 50/198 ... yes 136/199 ... yes 321/201 ... yes 2/202 ... yes 321/203 ... yes 2/207 ... yes 14/208 ... yes 65/213 ... yes 65/214 ... yes 3/216 ... yes 14/218 ... yes 66/219 ... yes 66/220 ... yes 32/221 ... yes 49/222 ... yes 50/223 ... yes 61/224 ... yes 110/226 ... yes 49/228 ... yes 49/229 ... yes 97/230 ... yes 41/232 ... yes 9/233 ... yes 49/234 ... yes 24/235 ... yes 69/236 ... yes 24/237 ... yes 24/238 ... yes 46/242 ... yes 46/243 ... yes 46/244 ... yes 46/245 ... yes 58/246 ... yes 62/247 ... yes 62/248 ... yes 330/250 ... yes 32/251 ... yes 50/253 ... yes 28/254 ... yes 50/256 ... yes 72/257 ... yes 42/258 ... yes 68/259 ... yes 59/260 ... yes 65/262 ... yes 75/263 ... yes 2/264 ... yes 2/265 ... yes 14/266 ... yes 48/267 ... yes 9/268 ... yes 41/269 ... yes 28/270 ... yes 75/271 ... yes 9/272 ... yes 71/273 ... yes 71/274 ... yes 80/275 ... yes 82/276 ... yes 58/277 ... yes 92/278 ... yes 24/279 ... yes 99/280 ... yes 99/281 ... yes 100/282 ... yes 101/283 ... yes 84/284 ... yes 102/285 ... yes 325/289 ... yes 103/290 ... yes 104/291 ... yes 104/292 ... yes 88/294 ... yes 50/295 ... yes 58/296 ... yes 66/297 ... yes 107/299 ... yes 108/300 ... yes 86/301 ... yes 24/302 ... yes 89/303 ... yes 109/304 ... yes 9/305 ... yes 322/306 ... yes 322/307 ... yes 135/308 ... yes 65/309 ... yes 79/310 ... yes 66/312 ... yes 85/313 ... yes 111/314 ... yes 246/315 ... yes 112/316 ... yes 322/317 ... yes 41/318 ... yes 114/319 ... yes 115/320 ... yes 116/321 ... yes 117/322 ... yes 118/323 ... yes 79/328 ... yes 119/329 ... yes 2/330 ... yes 79/331 ... yes 79/332 ... yes 138/334 ... yes 96/335 ... yes 133/336 ... yes 122/337 ... yes 8/338 ... yes 123/339 ... yes 126/341 ... yes 66/342 ... yes 127/343 ... yes 9/344 ... yes 130/346 ... yes 110/347 ... yes 327/348 ... yes 4/349 ... yes 138/350 ... yes 138/351 ... yes 325/352 ... yes 66/353 ... yes 131/354 ... yes 17/355 ... yes 132/356 ... yes 64/357 ... yes 2/359 ... yes 65/360 ... yes 327/361 ... yes 65/362 ... yes 143/363 ... yes 144/364 ... yes 144/365 ... yes 147/367 ... yes 147/369 ... yes 148/370 ... yes 149/371 ... yes 327/372 ... yes 149/373 ... yes 150/374 ... yes 153/376 ... yes 154/378 ... yes 156/380 ... yes 158/382 ... yes 158/383 ... yes 145/384 ... yes 159/385 ... yes 146/386 ... yes 160/387 ... yes 151/388 ... yes 161/389 ... yes 155/390 ... yes 162/391 ... yes 164/392 ... yes 163/393 ... yes 165/394 ... yes 165/395 ... yes 166/396 ... yes 277/397 ... yes 166/399 ... yes 169/400 ... yes 169/401 ... yes 170/402 ... yes 162/403 ... yes 146/404 ... yes 90/405 ... yes 65/406 ... yes 65/407 ... yes 322/408 ... yes 65/409 ... yes 50/410 ... yes 2/411 ... yes 173/412 ... yes 50/413 ... yes 178/414 ... yes 65/415 ... yes 17/417 ... yes 17/418 ... yes 4/419 ... yes 181/420 ... yes 138/421 ... yes 182/422 ... yes 324/423 ... yes 183/424 ... yes 138/425 ... yes 66/426 ... yes 46/427 ... yes 184/428 ... yes 37/429 ... yes 14/430 ... yes 324/431 ... yes 185/432 ... yes 186/433 ... yes 41/434 ... yes 330/435 ... yes 189/436 ... yes 190/437 ... yes 191/438 ... yes 17/439 ... yes 32/440 ... yes 192/441 ... yes 108/442 ... yes 50/444 ... yes 328/445 ... yes 2/446 ... yes 328/447 ... yes 320/448 ... yes 50/449 ... yes 195/450 ... yes 65/451 ... yes 9/452 ... yes 390/453 ... yes 390/454 ... yes 199/455 ... yes 148/456 ... yes 184/457 ... yes 196/458 ... yes 66/459 ... yes 200/461 ... yes 65/462 ... yes 184/463 ... yes 330/464 ... yes 9/465 ... yes 202/466 ... yes 320/467 ... yes 120/468 ... yes 225/469 ... yes 9/470 ... yes 202/471 ... yes 3/472 ... yes 390/473 ... yes 390/474 ... yes 323/475 ... yes 9/476 ... yes 32/477 ... yes 19/478 ... yes 2/479 ... yes 3/480 ... yes 130/481 ... yes 206/482 ... yes 32/483 ... yes 2/484 ... yes 320/485 ... yes 320/486 ... yes 2/487 ... yes 355/489 ... yes 320/492 ... yes 323/493 ... yes 138/494 ... yes 138/495 ... yes 191/496 ... yes 191/498 ... yes 9/499 ... yes 320/500 ... yes 295/502 ... yes 211/504 ... yes 200/505 ... yes 330/506 ... yes 32/507 ... yes 209/508 ... yes 9/509 ... yes 79/510 ... yes 65/511 ... yes 204/512 ... yes 322/513 ... yes 213/514 ... yes 214/516 ... yes 295/517 ... yes 58/518 ... yes 326/519 ... yes 191/520 ... yes 328/522 ... yes 322/523 ... yes 217/524 ... yes 120/525 ... yes 320/526 ... yes 330/527 ... yes 330/528 ... yes 211/529 ... yes 248/530 ... yes 221/531 ... yes 222/532 ... yes 223/533 ... yes 323/534 ... yes 219/535 ... yes 28/536 ... yes 65/537 ... yes 221/538 ... yes 225/540 ... yes 225/541 ... yes 196/542 ... yes 138/543 ... yes 226/544 ... yes 28/545 ... yes 221/546 ... yes 28/547 ... yes 28/548 ... yes 24/550 ... yes 55/551 ... yes 9/552 ... yes 295/553 ... yes 320/555 ... yes 204/556 ... yes 221/557 ... yes 228/559 ... yes 329/560 ... yes 221/561 ... yes 65/562 ... yes 144/563 ... yes 329/564 ... yes 9/566 ... yes 204/568 ... yes 65/569 ... yes 231/570 ... yes 65/571 ... yes 232/572 ... yes 390/573 ... yes 320/574 ... yes 320/575 ... yes 50/576 ... yes 65/577 ... yes 92/578 ... yes 65/579 ... yes 184/581 ... yes 233/582 ... yes 233/583 ... yes 233/584 ... yes 234/585 ... yes 234/586 ... yes 65/590 ... yes 65/591 ... yes 221/592 ... yes 390/593 ... yes 19/596 ... yes 17/597 ... yes 237/598 ... yes 221/600 ... yes 65/601 ... yes 221/602 ... yes 221/603 ... yes 238/604 ... yes 242/605 ... yes 221/606 ... yes 234/607 ... yes 237/608 ... yes 246/609 ... yes 243/610 ... yes 245/611 ... yes 214/612 ... yes 19/613 ... yes 152/615 ... yes 27/616 ... yes 211/617 ... yes 19/618 ... yes 8/621 ... yes 106/622 ... yes 2/623 ... yes 204/624 ... yes 225/625 ... yes 238/627 ... yes 32/628 ... yes 322/630 ... yes 32/631 ... yes 217/632 ... yes 221/633 ... yes 261/634 ... yes 262/635 ... yes 90/636 ... yes 90/637 ... yes 268/638 ... yes 274/639 ... yes 225/640 ... yes 91/641 ... yes 150/642 ... yes 277/643 ... yes 268/644 ... yes 32/645 ... yes 2/647 ... yes 286/648 ... yes 283/649 ... yes 217/650 ... yes 2/651 ... yes 92/652 ... yes 92/653 ... yes 92/654 ... yes 92/655 ... yes 92/656 ... yes 92/657 ... yes 32/658 ... yes 32/659 ... yes 37/660 ... yes 66/661 ... yes 237/665 ... yes 213/666 ... yes 37/667 ... yes 211/668 ... yes 300/670 ... yes 300/671 ... yes 300/672 ... yes 211/673 ... yes 200/675 ... yes 37/676 ... yes 221/677 ... yes 305/678 ... yes 9/679 ... yes 147/680 ... yes 147/681 ... yes 147/683 ... yes 2/684 ... yes 147/685 ... yes 309/687 ... yes 8/688 ... yes 313/689 ... yes 64/690 ... yes 331/693 ... yes 320/694 ... yes 92/696 ... yes 332/697 ... yes 331/698 ... yes 305/699 ... yes 305/700 ... yes 37/702 ... yes 274/703 ... yes 330/704 ... yes 196/705 ... yes 2/706 ... yes 2/707 ... yes 286/708 ... yes 238/709 ... yes 211/710 ... yes 211/711 ... yes 209/712 ... yes 354/714 ... yes 309/715 ... yes 32/716 ... yes 66/717 ... yes 50/718 ... yes 352/719 ... yes 286/720 ... yes 41/721 ... yes 353/722 ... yes 353/723 ... yes 92/724 ... yes 92/725 ... yes 277/726 ... yes 211/727 ... yes 262/728 ... yes 353/729 ... yes 353/730 ... yes 129/731 ... yes 92/732 ... yes 221/733 ... yes 274/735 ... yes 238/736 ... yes 355/737 ... yes 234/738 ... yes 305/739 ... yes 184/740 ... yes 154/741 ... yes 202/742 ... yes 358/743 ... yes 211/744 ... yes 50/745 ... yes 211/746 ... yes 262/748 ... yes 73/749 ... yes 4/750 ... yes 154/751 ... yes 115/752 ... yes 19/753 ... yes 19/754 ... yes 32/755 ... yes 154/756 ... yes 262/757 ... yes 234/758 ... yes 66/759 ... yes 274/760 ... yes 40/761 ... yes 106/762 ... yes 329/763 ... yes 366/764 ... yes 370/765 ... yes 233/766 ... yes 330/767 ... yes 390/768 ... yes 94/771 ... yes 94/772 ... yes 246/773 ... yes 320/774 ... yes 66/775 ... yes 377/776 ... yes 32/777 ... yes 196/778 ... yes 295/779 ... yes 206/780 ... yes 330/781 ... yes 320/782 ... yes 66/783 ... yes 2/784 ... yes 295/785 ... yes 295/786 ... yes 184/787 ... yes 383/788 ... yes 24/789 ... yes 377/790 ... yes 50/791 ... yes 389/792 ... yes 295/793 ... yes 320/794 ... yes 392/795 ... yes 9/796 ... yes 184/797 ... yes 237/798 ... yes 295/799 ... yes 326/800 ... yes 396/801 ... yes 396/802 ... yes 396/803 ... yes 154/804 ... yes 402/807 ... yes 370/809 ... yes 370/810 ... yes 46/811 ... yes 246/812 ... yes 407/814 ... yes 370/815 ... yes 407/816 ... yes 407/817 ... yes 409/818 ... yes 64/819 ... yes 237/820 ... yes 32/821 ... yes 89/822 ... yes 330/823 ... yes 225/824 ... yes 320/826 ... yes 209/827 ... yes 392/830 ... yes 211/832 ... yes 27/833 ... yes 366/834 ... yes 366/837 ... yes 211/838 ... yes 144/839 ... yes 237/840 ... yes 300/842 ... yes 421/843 ... yes 421/844 ... yes 326/845 ... yes 423/846 ... yes 427/848 ... yes 65/850 ... yes 221/851 ... yes 24/852 ... yes 13/853 ... yes 221/854 ... yes 433/855 ... yes 225/856 ... yes 9/857 ... yes 163/858 ... yes 435/859 ... yes 286/860 ... yes 286/861 ... yes 232/863 ... yes 326/864 ... yes 66/865 ... yes 66/866 ... yes 17/867 ... yes 28/868 ... yes 196/869 ... yes 28/870 ... yes 246/871 ... yes 246/872 ... yes 65/873 ... yes 28/874 ... yes 246/875 ... yes 99/876 ... yes 9/877 ... yes 267/878 ... yes 449/879 ... yes 203/880 ... yes 238/881 ... yes 17/882 ... yes 221/883 ... yes 65/884 ... yes 50/885 ... yes 221/886 ... yes 246/887 ... yes 66/889 ... yes 28/890 ... yes 221/891 ... yes 108/892 ... yes 246/893 ... yes 366/894 ... yes 471/895 ... yes 472/896 ... yes 203/897 ... yes 277/898 ... yes 246/899 ... yes 221/900 ... yes 211/901 ... yes 99/902 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.5) Git version >= 2.22.0 ? ... yes (2.24.1) Git user has default SSH configuration? ... yes Active users: ... 216 Is authorized keys file accessible? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished