ReactiveCaching sidekiq queue overloaded (TeamCity)
Summary
When a (TeamCity) server listed in an integration is unreachable, the Sidekiq reactive_caching queue fills with more and more jobs, which impact GitLab performance after a while.
Steps to reproduce
- Configure TeamCity integration on (multiple?) projects.
- Stop TeamCity server
- Wait some hours
Example Project
None.
What is the current bug behavior?
As visible in https://SERVER/admin/sidekiq/queues/reactive_caching, the queue fills with ReactiveCachingWorker jobs with the arguments "TeamcityService", , "some-hex-string", "branch-name", all failing with
Errno::ECONNREFUSED: Failed to open TCP connection to teamcity:80 (Connection refused - connect(2) for "teamcity" port 80)
What is the expected correct behavior?
I would expact at most one ReactiveCachingWorker job per GitLab project (maybe per branch) - or (if there are multiple jobs per project) with a limited number of retries or some kind of back-off behaviour, maybe even automatically disabling the integration if the problem persists over a prolonged period.
Relevant logs and/or screenshots
from sidekiq.log:
2019-07-02_08:33:19.70529 2019-07-02T08:33:19.616Z 18189 TID-owfswkn8t WARN: {"context":"Job raised exception","job":{"class":"ReactiveCachingWorker","args":["TeamcityService",999,"XXX","XXX"],"retry":3,"queue":"reactive_caching","jid":"XXX","created_at":1562055403.3611827,"correlation_id":"XXX","enqueued_at":1562056399.5439627},"jobstr":"{\"class\":\"ReactiveCachingWorker\",\"args\":[\"TeamcityService\",999,\"XXX\",\"XXX\"],\"retry\":3,\"queue\":\"reactive_caching\",\"jid\":\"XXX\",\"created_at\":1562055403.3611827,\"correlation_id\":\"XXX\",\"enqueued_at\":1562056399.5439627}"}
2019-07-02_08:33:19.70530 2019-07-02T08:33:19.616Z 18189 TID-owfswkn8t WARN: Errno::ECONNREFUSED: Failed to open TCP connection to teamcity:80 (Connection refused - connect(2) for "teamcity" port 80)
2019-07-02_08:33:19.70532 2019-07-02T08:33:19.616Z 18189 TID-owfswkn8t WARN: /opt/gitlab/embedded/lib/ruby/2.5.0/net/http.rb:939:in `rescue in block in connect'
2019-07-02_08:33:19.70532 /opt/gitlab/embedded/lib/ruby/2.5.0/net/http.rb:936:in `block in connect'
2019-07-02_08:33:19.70533 /opt/gitlab/embedded/lib/ruby/2.5.0/timeout.rb:93:in `block in timeout'
2019-07-02_08:33:19.70534 /opt/gitlab/embedded/lib/ruby/2.5.0/timeout.rb:103:in `timeout'
2019-07-02_08:33:19.70534 /opt/gitlab/embedded/lib/ruby/2.5.0/net/http.rb:935:in `connect'
2019-07-02_08:33:19.70535 /opt/gitlab/embedded/lib/ruby/2.5.0/net/http.rb:920:in `do_start'
2019-07-02_08:33:19.70535 /opt/gitlab/embedded/lib/ruby/2.5.0/net/http.rb:909:in `start'
2019-07-02_08:33:19.70536 /opt/gitlab/embedded/lib/ruby/2.5.0/net/http.rb:1455:in `request'
2019-07-02_08:33:19.70536 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httparty-0.13.7/lib/httparty/request.rb:117:in `perform'
2019-07-02_08:33:19.70537 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httparty-0.13.7/lib/httparty.rb:545:in `perform_request'
2019-07-02_08:33:19.70537 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/http.rb:17:in `perform_request'
2019-07-02_08:33:19.70538 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/httparty-0.13.7/lib/httparty.rb:476:in `get'
2019-07-02_08:33:19.70540 /opt/gitlab/embedded/service/gitlab-rails/app/models/project_services/teamcity_service.rb:137:in `get_path'
2019-07-02_08:33:19.70540 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:161:in `block in get_path'
2019-07-02_08:33:19.70541 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/method_call.rb:36:in `measure'
2019-07-02_08:33:19.70541 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:161:in `get_path'
2019-07-02_08:33:19.70543 /opt/gitlab/embedded/service/gitlab-rails/app/models/project_services/teamcity_service.rb:71:in `calculate_reactive_cache'
2019-07-02_08:33:19.70543 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:161:in `block in calculate_reactive_cache'
2019-07-02_08:33:19.70544 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/method_call.rb:36:in `measure'
2019-07-02_08:33:19.70544 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:161:in `calculate_reactive_cache'
2019-07-02_08:33:19.70545 /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/reactive_caching.rb:96:in `block (2 levels) in exclusively_update_reactive_cache!'
2019-07-02_08:33:19.70547 /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/reactive_caching.rb:141:in `enqueuing_update'
2019-07-02_08:33:19.70547 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:161:in `block in enqueuing_update'
2019-07-02_08:33:19.70548 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/method_call.rb:36:in `measure'
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Debian 8.11 Proxy: no Current User: git Using RVM: no Ruby Version: 2.5.3p105 Gem Version: 2.7.6 Bundler Version:1.16.6 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.18.1 Sidekiq Version:5.2.5 Go Version: unknown
GitLab information Version: 11.9.4-ee Revision: 55be7f0 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql DB Version: 9.6.13 URL: https://internal HTTP Clone URL: https://internal/some-group/some-project.git SSH Clone URL: git@internal:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: yes Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 8.7.1 Repository storage paths:
- default: /mnt/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 8.7.1 ? ... OK (8.7.1) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) NOPE Server: ldapsecondary LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) NOPE
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 1/1 ... yes 4/2 ... yes 1/4 ... yes 1/5 ... yes 1/6 ... yes 1/7 ... yes 2/8 ... yes 7/9 ... yes 149/11 ... yes 1/12 ... yes 1/13 ... yes 1/14 ... yes 7/15 ... yes 448/17 ... yes 663/18 ... yes 449/21 ... yes 18/25 ... yes 149/26 ... yes 21/43 ... yes 2/45 ... yes 27/57 ... yes 27/58 ... yes 27/59 ... yes 449/60 ... yes 449/61 ... yes 10/62 ... yes 450/63 ... yes 450/64 ... yes 10/65 ... yes 10/66 ... yes 10/68 ... yes 10/69 ... yes 171/70 ... yes 29/71 ... yes 11/72 ... yes 11/73 ... yes 171/74 ... yes 11/75 ... yes 11/76 ... yes 11/77 ... yes 11/78 ... yes 11/79 ... yes 31/80 ... yes 18/82 ... yes 18/83 ... yes 18/84 ... yes 18/85 ... yes 18/86 ... yes 18/87 ... yes 18/88 ... yes 18/89 ... yes 38/90 ... yes 38/93 ... yes 41/94 ... yes 292/98 ... yes 72/100 ... yes 72/101 ... yes 72/102 ... yes 271/103 ... yes 76/108 ... yes 76/109 ... yes 14/113 ... yes 41/115 ... yes 41/117 ... yes 76/118 ... yes 299/121 ... yes 3/122 ... yes 91/123 ... yes 91/124 ... yes 91/125 ... yes 75/126 ... yes 637/129 ... yes 637/130 ... yes 637/131 ... yes 637/132 ... yes 637/133 ... yes 637/134 ... yes 637/135 ... yes 637/136 ... yes 637/137 ... yes 637/138 ... yes 637/139 ... yes 637/140 ... yes 637/141 ... yes 637/142 ... yes 637/143 ... yes 637/144 ... yes 637/145 ... yes 637/146 ... yes 637/148 ... yes 637/149 ... yes 637/150 ... yes 637/151 ... yes 637/152 ... yes 637/153 ... yes 637/154 ... yes 637/155 ... yes 637/156 ... yes 637/157 ... yes 637/158 ... yes 637/159 ... yes 637/160 ... yes 637/161 ... yes 75/166 ... yes 75/167 ... yes 75/168 ... yes 75/169 ... yes 75/170 ... yes 75/171 ... yes 41/172 ... yes 650/174 ... yes 3/175 ... yes 294/176 ... yes 294/178 ... yes 292/179 ... yes 38/180 ... yes 38/181 ... yes 38/182 ... yes 38/183 ... yes 2/184 ... yes 2/185 ... yes 2/186 ... yes 49/187 ... yes 286/188 ... yes 286/189 ... yes 286/190 ... yes 663/191 ... yes 74/192 ... yes 294/198 ... yes 41/200 ... yes 38/201 ... yes 2/202 ... yes 2/203 ... yes 24/204 ... yes 12/205 ... yes 40/206 ... yes 41/207 ... yes 120/210 ... yes 113/211 ... yes 6/217 ... yes 11/222 ... yes 6/223 ... yes 91/225 ... yes 12/226 ... yes 29/230 ... yes 29/232 ... yes 11/233 ... yes 31/235 ... yes 41/236 ... yes 27/238 ... yes 129/239 ... yes 663/240 ... yes 120/241 ... yes 41/242 ... yes 29/244 ... yes 29/245 ... yes 322/246 ... yes 29/247 ... yes 322/248 ... yes 322/250 ... yes 29/251 ... yes 131/253 ... yes 131/254 ... yes 131/255 ... yes 2/258 ... yes 113/259 ... yes 2/260 ... yes 2/261 ... yes 12/262 ... yes 72/263 ... yes 41/264 ... yes 294/265 ... yes 12/268 ... yes 157/269 ... yes 289/270 ... yes 12/271 ... yes 294/272 ... yes 12/273 ... yes 138/275 ... yes 138/276 ... yes 138/277 ... yes 138/278 ... yes 138/279 ... yes 138/280 ... yes 138/281 ... yes 138/282 ... yes 138/283 ... yes 138/284 ... yes 138/285 ... yes 138/286 ... yes 490/288 ... yes 11/289 ... yes 11/290 ... yes 24/291 ... yes 78/292 ... yes 78/293 ... yes 78/294 ... yes 83/296 ... yes 31/297 ... yes 140/298 ... yes 41/299 ... yes 140/302 ... yes 2/303 ... yes 286/304 ... yes 144/306 ... yes 144/308 ... yes 146/309 ... yes 650/310 ... yes 33/313 ... yes 23/314 ... yes 23/316 ... yes 23/317 ... yes 146/318 ... yes 547/319 ... yes 548/320 ... yes 14/323 ... yes 14/324 ... yes 14/325 ... yes 144/326 ... yes 2/327 ... yes 2/328 ... yes 648/329 ... yes 650/330 ... yes 637/332 ... yes 2/333 ... yes 294/334 ... yes 27/335 ... yes 299/336 ... yes 153/337 ... yes 18/338 ... yes 153/342 ... yes 294/344 ... yes 27/347 ... yes 38/348 ... yes 5/349 ... yes 83/353 ... yes 153/355 ... yes 153/356 ... yes 153/357 ... yes 474/358 ... yes 12/359 ... yes 5/360 ... yes 29/361 ... yes 38/362 ... yes 5/363 ... yes 5/364 ... yes 5/365 ... yes 29/366 ... yes 29/367 ... yes 18/368 ... yes 158/372 ... yes 103/374 ... yes 42/379 ... yes 598/380 ... yes 42/384 ... yes 105/385 ... yes 10/386 ... yes 532/390 ... yes 157/391 ... yes 42/392 ... yes 448/393 ... yes 38/395 ... yes 14/398 ... yes 371/399 ... yes 371/400 ... yes 29/401 ... yes 637/402 ... yes 113/404 ... yes 146/405 ... yes 132/407 ... yes 14/409 ... yes 11/410 ... yes 88/411 ... yes 295/412 ... yes 41/414 ... yes 42/415 ... yes 17/416 ... yes 17/417 ... yes 14/420 ... yes 294/421 ... yes 14/422 ... yes 38/423 ... yes 120/426 ... yes 83/427 ... yes 140/428 ... yes 38/429 ... yes 120/430 ... yes 129/431 ... yes 356/434 ... yes 343/436 ... yes 27/438 ... yes 322/439 ... yes 158/440 ... yes 125/441 ... yes 356/442 ... yes 356/443 ... yes 5/444 ... yes 182/446 ... yes 294/447 ... yes 144/448 ... yes 10/450 ... yes 120/451 ... yes 175/452 ... yes 145/453 ... yes 3/457 ... yes 138/458 ... yes 79/460 ... yes 182/461 ... yes 41/463 ... yes 41/465 ... yes 149/466 ... yes 38/471 ... yes 138/472 ... yes 48/474 ... yes 9/475 ... yes 11/477 ... yes 192/479 ... yes 181/481 ... yes 181/482 ... yes 181/483 ... yes 12/484 ... yes 101/485 ... yes 41/486 ... yes 138/490 ... yes 31/491 ... yes 181/493 ... yes 38/495 ... yes 160/498 ... yes 299/499 ... yes 83/500 ... yes 637/502 ... yes 138/503 ... yes 138/504 ... yes 14/505 ... yes 83/507 ... yes 5/508 ... yes 294/510 ... yes 47/511 ... yes 639/512 ... yes 129/513 ... yes 138/514 ... yes 38/515 ... yes 138/517 ... yes 129/518 ... yes 120/519 ... yes 157/520 ... yes 21/521 ... yes 344/522 ... yes 344/523 ... yes 181/524 ... yes 27/525 ... yes 9/526 ... yes 175/527 ... yes 129/528 ... yes 299/530 ... yes 31/532 ... yes 132/533 ... yes 31/538 ... yes 113/540 ... yes 31/541 ... yes 138/542 ... yes 18/543 ... yes 175/544 ... yes 18/545 ... yes 18/546 ... yes 18/547 ... yes 18/548 ... yes 18/549 ... yes 294/550 ... yes 181/554 ... yes 184/555 ... yes 181/556 ... yes 183/557 ... yes 120/561 ... yes 120/562 ... yes 31/563 ... yes 286/564 ... yes 31/566 ... yes 31/567 ... yes 5/570 ... yes 138/571 ... yes 11/573 ... yes 663/574 ... yes 38/576 ... yes 356/579 ... yes 299/580 ... yes 294/581 ... yes 294/582 ... yes 322/583 ... yes 5/584 ... yes 18/585 ... yes 65/586 ... yes 138/592 ... yes 138/594 ... yes 356/596 ... yes 138/597 ... yes 129/598 ... yes 120/600 ... yes 75/601 ... yes 158/602 ... yes 31/603 ... yes 129/604 ... yes 5/605 ... yes 5/606 ... yes 5/607 ... yes 299/608 ... yes 299/609 ... yes 221/610 ... yes 27/611 ... yes 140/612 ... yes 214/613 ... yes 42/614 ... yes 534/615 ... yes 83/616 ... yes 13/619 ... yes 343/621 ... yes 299/622 ... yes 31/623 ... yes 141/624 ... yes 286/625 ... yes 181/628 ... yes 83/629 ... yes 532/631 ... yes 83/632 ... yes 593/634 ... yes 38/635 ... yes 181/636 ... yes 46/637 ... yes 29/639 ... yes 175/640 ... yes 229/642 ... yes 83/643 ... yes 4/644 ... yes 12/645 ... yes 181/651 ... yes 181/652 ... yes 181/655 ... yes 83/656 ... yes 129/658 ... yes 175/661 ... yes 113/663 ... yes 650/665 ... yes 83/666 ... yes 532/667 ... yes 113/670 ... yes 125/671 ... yes 125/672 ... yes 322/673 ... yes 113/674 ... yes 5/675 ... yes 160/676 ... yes 175/677 ... yes 175/678 ... yes 158/679 ... yes 29/682 ... yes 158/686 ... yes 29/687 ... yes 129/690 ... yes 158/692 ... yes 295/693 ... yes 5/694 ... yes 72/695 ... yes 283/696 ... yes 129/697 ... yes 294/698 ... yes 122/699 ... yes 113/700 ... yes 663/702 ... yes 4/703 ... yes 5/704 ... yes 294/705 ... yes 181/706 ... yes 286/707 ... yes 182/708 ... yes 251/711 ... yes 251/716 ... yes 283/717 ... yes 38/718 ... yes 283/720 ... yes 251/722 ... yes 103/723 ... yes 29/724 ... yes 70/725 ... yes 29/726 ... yes 10/727 ... yes 27/730 ... yes 41/731 ... yes 175/732 ... yes 83/733 ... yes 13/734 ... yes 13/735 ... yes 120/736 ... yes 246/737 ... yes 650/738 ... yes 132/740 ... yes 132/741 ... yes 292/743 ... yes 251/745 ... yes 246/746 ... yes 29/747 ... yes 271/750 ... yes 271/751 ... yes 271/752 ... yes 283/753 ... yes 83/755 ... yes 283/756 ... yes 489/757 ... yes 271/758 ... yes 271/759 ... yes 271/760 ... yes 271/761 ... yes 276/762 ... yes 272/763 ... yes 272/764 ... yes 272/765 ... yes 122/766 ... yes 275/767 ... yes 276/768 ... yes 276/769 ... yes 276/770 ... yes 276/771 ... yes 276/772 ... yes 276/773 ... yes 276/774 ... yes 276/775 ... yes 273/778 ... yes 273/779 ... yes 273/780 ... yes 273/781 ... yes 72/782 ... yes 72/784 ... yes 261/785 ... yes 261/786 ... yes 35/787 ... yes 5/788 ... yes 72/789 ... yes 274/791 ... yes 273/792 ... yes 192/793 ... yes 38/796 ... yes 192/799 ... yes 40/800 ... yes 577/802 ... yes 283/803 ... yes 251/808 ... yes 132/809 ... yes 42/810 ... yes 143/811 ... yes 143/812 ... yes 64/813 ... yes 287/814 ... yes 138/816 ... yes 292/817 ... yes 132/819 ... yes 132/820 ... yes 181/826 ... yes 12/827 ... yes 12/829 ... yes 83/830 ... yes 12/831 ... yes 250/835 ... yes 175/837 ... yes 271/838 ... yes 261/839 ... yes 83/840 ... yes 40/841 ... yes 255/842 ... yes 257/843 ... yes 172/844 ... yes 299/845 ... yes 663/846 ... yes 17/848 ... yes 181/849 ... yes 257/850 ... yes 83/851 ... yes 83/852 ... yes 637/853 ... yes 144/854 ... yes 172/855 ... yes 255/856 ... yes 83/857 ... yes 65/858 ... yes 226/859 ... yes 226/860 ... yes 226/861 ... yes 297/862 ... yes 308/863 ... yes 297/864 ... yes 83/868 ... yes 308/869 ... yes 311/870 ... yes 27/872 ... yes 294/873 ... yes 663/874 ... yes 314/877 ... yes 531/878 ... yes 42/879 ... yes 13/880 ... yes 315/881 ... yes 295/883 ... yes 287/885 ... yes 132/886 ... yes 294/888 ... yes 648/889 ... yes 648/890 ... yes 637/891 ... yes 637/892 ... yes 637/893 ... yes 294/894 ... yes 317/896 ... yes 12/898 ... yes 319/899 ... yes 661/902 ... yes 317/904 ... yes 330/906 ... yes 351/907 ... yes 322/908 ... yes 221/909 ... yes 247/910 ... yes 83/912 ... yes 287/913 ... yes 283/915 ... yes 283/916 ... yes 83/917 ... yes 20/918 ... yes 42/921 ... yes 663/922 ... yes 283/923 ... yes 283/924 ... yes 283/925 ... yes 335/926 ... yes 20/927 ... yes 158/928 ... yes 27/929 ... yes 299/930 ... yes 402/931 ... yes 179/932 ... yes 663/933 ... yes 339/934 ... yes 339/935 ... yes 20/936 ... yes 47/938 ... yes 181/939 ... yes 341/943 ... yes 343/945 ... yes 72/946 ... yes 648/947 ... yes 138/948 ... yes 351/949 ... yes 283/950 ... yes 287/952 ... yes 181/953 ... yes 343/954 ... yes 356/955 ... yes 351/956 ... yes 351/957 ... yes 351/958 ... yes 49/959 ... yes 531/960 ... yes 175/961 ... yes 322/962 ... yes 299/963 ... yes 284/964 ... yes 489/965 ... yes 212/967 ... yes 341/970 ... yes 225/972 ... yes 295/973 ... yes 311/974 ... yes 320/975 ... yes 311/976 ... yes 329/980 ... yes 357/982 ... yes 357/983 ... yes 256/984 ... yes 27/986 ... yes 24/987 ... yes 83/988 ... yes 294/989 ... yes 20/990 ... yes 70/991 ... yes 326/992 ... yes 637/994 ... yes 351/995 ... yes 335/996 ... yes 138/997 ... yes 370/998 ... yes 370/999 ... yes 341/1000 ... yes 160/1001 ... yes 322/1002 ... yes 351/1003 ... yes 42/1005 ... yes 256/1006 ... yes 320/1008 ... yes 356/1010 ... yes 650/1011 ... yes 83/1013 ... yes 324/1014 ... yes 387/1015 ... yes 83/1016 ... yes 390/1018 ... yes 47/1019 ... yes 388/1020 ... yes 391/1021 ... yes 393/1023 ... yes 283/1024 ... yes 394/1026 ... yes 357/1028 ... yes 299/1029 ... yes 283/1030 ... yes 351/1031 ... yes 663/1032 ... yes 131/1035 ... yes 532/1036 ... yes 83/1037 ... yes 392/1038 ... yes 392/1039 ... yes 618/1040 ... yes 20/1041 ... yes 391/1042 ... yes 329/1043 ... yes 289/1044 ... yes 283/1052 ... yes 125/1053 ... yes 322/1054 ... yes 322/1055 ... yes 18/1057 ... yes 13/1058 ... yes 406/1060 ... yes 322/1063 ... yes 322/1064 ... yes 351/1065 ... yes 292/1066 ... yes 103/1067 ... yes 13/1068 ... yes 295/1069 ... yes 286/1070 ... yes 663/1072 ... yes 292/1073 ... yes 322/1074 ... yes 283/1075 ... yes 297/1076 ... yes 83/1077 ... yes 329/1078 ... yes 250/1080 ... yes 13/1081 ... yes 297/1082 ... yes 283/1083 ... yes 299/1087 ... yes 299/1088 ... yes 299/1089 ... yes 283/1090 ... yes 349/1094 ... yes 349/1095 ... yes 18/1097 ... yes 257/1101 ... yes 294/1102 ... yes 261/1103 ... yes 45/1106 ... yes 289/1107 ... yes 418/1110 ... yes 663/1111 ... yes 292/1112 ... yes 138/1113 ... yes 138/1114 ... yes 344/1115 ... yes 29/1116 ... yes 103/1117 ... yes 287/1118 ... yes 423/1120 ... yes 181/1121 ... yes 667/1122 ... yes 132/1127 ... yes 172/1128 ... yes 132/1129 ... yes 637/1130 ... yes 283/1133 ... yes 12/1134 ... yes 12/1135 ... yes 373/1136 ... yes 4/1137 ... yes 132/1138 ... yes 29/1139 ... yes 42/1140 ... yes 432/1142 ... yes 292/1143 ... yes 432/1144 ... yes 145/1145 ... yes 322/1146 ... yes 294/1147 ... yes 432/1148 ... yes 160/1149 ... yes 76/1150 ... yes 18/1151 ... yes 388/1152 ... yes 343/1153 ... yes 447/1154 ... yes 40/1156 ... yes 261/1157 ... yes 261/1158 ... yes 279/1159 ... yes 447/1160 ... yes 489/1161 ... yes 192/1164 ... yes 33/1165 ... yes 12/1166 ... yes 192/1167 ... yes 192/1169 ... yes 437/1170 ... yes 261/1172 ... yes 341/1173 ... yes 447/1175 ... yes 195/1176 ... yes 663/1177 ... yes 297/1178 ... yes 83/1179 ... yes 439/1180 ... yes 12/1183 ... yes 32/1184 ... yes 442/1187 ... yes 83/1188 ... yes 190/1189 ... yes 442/1190 ... yes 125/1192 ... yes 141/1194 ... yes 343/1195 ... yes 162/1196 ... yes 195/1197 ... yes 13/1198 ... yes 182/1199 ... yes 447/1200 ... yes 294/1201 ... yes 60/1202 ... yes 181/1203 ... yes 45/1204 ... yes 294/1205 ... yes 447/1206 ... yes 192/1208 ... yes 160/1209 ... yes 326/1212 ... yes 181/1214 ... yes 181/1215 ... yes 447/1216 ... yes 322/1218 ... yes 342/1220 ... yes 35/1221 ... yes 326/1222 ... yes 283/1223 ... yes 447/1224 ... yes 295/1225 ... yes 18/1227 ... yes 287/1228 ... yes 18/1229 ... yes 432/1230 ... yes 283/1231 ... yes 283/1232 ... yes 283/1233 ... yes 299/1239 ... yes 391/1240 ... yes 292/1241 ... yes 160/1243 ... yes 299/1244 ... yes 343/1247 ... yes 283/1248 ... yes 192/1249 ... yes 283/1250 ... yes 440/1251 ... yes 45/1252 ... yes 447/1253 ... yes 448/1255 ... yes 449/1256 ... yes 356/1258 ... yes 295/1259 ... yes 181/1261 ... yes 4/1262 ... yes 144/1263 ... yes 144/1264 ... yes 144/1265 ... yes 365/1266 ... yes 365/1267 ... yes 181/1268 ... yes 289/1269 ... yes 664/1270 ... yes 212/1271 ... yes 88/1272 ... yes 593/1273 ... yes 138/1276 ... yes 160/1277 ... yes 660/1278 ... yes 286/1279 ... yes 472/1280 ... yes 166/1281 ... yes 53/1282 ... yes 340/1283 ... yes 312/1284 ... yes 45/1285 ... yes 295/1286 ... yes 478/1287 ... yes 125/1288 ... yes 40/1289 ... yes 478/1290 ... yes 398/1291 ... yes 398/1292 ... yes 299/1293 ... yes 286/1294 ... yes 398/1295 ... yes 439/1298 ... yes 474/1299 ... yes 13/1300 ... yes 338/1301 ... yes 254/1302 ... yes 192/1303 ... yes 322/1304 ... yes 314/1305 ... yes 27/1306 ... yes 341/1307 ... yes 341/1308 ... yes 125/1309 ... yes 289/1311 ... yes 246/1312 ... yes 492/1313 ... yes 492/1314 ... yes 45/1315 ... yes 492/1316 ... yes 478/1317 ... yes 289/1318 ... yes 491/1319 ... yes 493/1321 ... yes 493/1322 ... yes 495/1323 ... yes 12/1324 ... yes 78/1325 ... yes 490/1326 ... yes 503/1327 ... yes 503/1328 ... yes 503/1329 ... yes 274/1330 ... yes 144/1331 ... yes 144/1332 ... yes 144/1333 ... yes 144/1334 ... yes 144/1335 ... yes 144/1336 ... yes 341/1338 ... yes 192/1339 ... yes 668/1340 ... yes 510/1342 ... yes 292/1343 ... yes 322/1344 ... yes 322/1345 ... yes 101/1347 ... yes 398/1348 ... yes 138/1349 ... yes 287/1350 ... yes 73/1351 ... yes 512/1352 ... yes 343/1353 ... yes 294/1354 ... yes 27/1355 ... yes 492/1358 ... yes 65/1359 ... yes 343/1360 ... yes 40/1361 ... yes 440/1362 ... yes 453/1363 ... yes 418/1364 ... yes 144/1365 ... yes 144/1366 ... yes 190/1367 ... yes 491/1369 ... yes 181/1370 ... yes 190/1371 ... yes 190/1372 ... yes 190/1373 ... yes 387/1374 ... yes 287/1375 ... yes 295/1376 ... yes 343/1378 ... yes 485/1379 ... yes 493/1380 ... yes 531/1381 ... yes 531/1382 ... yes 294/1384 ... yes 320/1385 ... yes 320/1386 ... yes 287/1387 ... yes 440/1389 ... yes 478/1390 ... yes 549/1391 ... yes 549/1392 ... yes 549/1393 ... yes 549/1394 ... yes 549/1396 ... yes 33/1397 ... yes 293/1399 ... yes 341/1400 ... yes 322/1401 ... yes 461/1402 ... yes 322/1403 ... yes 549/1404 ... yes 666/1405 ... yes 190/1407 ... yes 76/1409 ... yes 532/1410 ... yes 305/1411 ... yes 364/1413 ... yes 47/1415 ... yes 649/1416 ... yes 47/1417 ... yes 637/1419 ... yes 138/1420 ... yes 392/1422 ... yes 568/1423 ... yes 579/1424 ... yes 532/1425 ... yes 565/1427 ... yes 565/1428 ... yes 565/1429 ... yes 565/1430 ... yes 565/1432 ... yes 565/1433 ... yes 565/1434 ... yes 565/1435 ... yes 565/1436 ... yes 565/1437 ... yes 565/1438 ... yes 565/1444 ... yes 565/1445 ... yes 479/1446 ... yes 565/1447 ... yes 565/1448 ... yes 565/1449 ... yes 565/1451 ... yes 540/1452 ... yes 581/1453 ... yes 292/1454 ... yes 567/1455 ... yes 567/1456 ... yes 569/1457 ... yes 568/1458 ... yes 570/1459 ... yes 570/1460 ... yes 289/1462 ... yes 402/1465 ... yes 565/1469 ... yes 65/1470 ... yes 649/1471 ... yes 649/1472 ... yes 649/1473 ... yes 649/1475 ... yes 649/1476 ... yes 565/1477 ... yes 565/1478 ... yes 47/1479 ... yes 565/1480 ... yes 565/1481 ... yes 173/1482 ... yes 365/1483 ... yes 452/1484 ... yes 47/1485 ... yes 649/1486 ... yes 580/1488 ... yes 596/1489 ... yes 287/1490 ... yes 579/1491 ... yes 192/1492 ... yes 592/1493 ... yes 592/1494 ... yes 592/1495 ... yes 577/1496 ... yes 577/1497 ... yes 592/1498 ... yes 591/1499 ... yes 591/1500 ... yes 365/1501 ... yes 418/1502 ... yes 596/1503 ... yes 566/1504 ... yes 592/1505 ... yes 294/1506 ... yes 103/1507 ... yes 593/1508 ... yes 27/1509 ... yes 602/1511 ... yes 591/1512 ... yes 507/1513 ... yes 591/1514 ... yes 591/1515 ... yes 591/1516 ... yes 591/1517 ... yes 591/1518 ... yes 591/1519 ... yes 591/1520 ... yes 591/1521 ... yes 591/1522 ... yes 591/1523 ... yes 591/1524 ... yes 286/1525 ... yes 591/1526 ... yes 591/1527 ... yes 591/1528 ... yes 209/1529 ... yes 343/1532 ... yes 320/1533 ... yes 144/1534 ... yes 591/1535 ... yes 591/1536 ... yes 579/1537 ... yes 343/1538 ... yes 591/1539 ... yes 342/1540 ... yes 447/1542 ... yes 442/1543 ... yes 335/1544 ... yes 605/1545 ... yes 591/1547 ... yes 591/1548 ... yes 649/1549 ... yes 418/1550 ... yes 447/1551 ... yes 294/1552 ... yes 418/1553 ... yes 27/1554 ... yes 294/1555 ... yes 611/1556 ... yes 611/1557 ... yes 655/1558 ... yes 611/1559 ... yes 612/1560 ... yes 612/1561 ... yes 276/1562 ... yes 276/1565 ... yes 276/1566 ... yes 276/1568 ... yes 27/1569 ... yes 61/1570 ... yes 292/1571 ... yes 611/1572 ... yes 173/1573 ... yes 398/1574 ... yes 276/1576 ... yes 276/1577 ... yes 387/1578 ... yes 398/1579 ... yes 398/1580 ... yes 287/1581 ... yes 276/1583 ... yes 611/1584 ... yes 250/1585 ... yes 250/1586 ... yes 250/1587 ... yes 27/1588 ... yes 250/1589 ... yes 299/1590 ... yes 398/1591 ... yes 173/1592 ... yes 398/1593 ... yes 398/1594 ... yes 491/1595 ... yes 617/1596 ... yes 398/1597 ... yes 493/1598 ... yes 398/1599 ... yes 442/1600 ... yes 343/1602 ... yes 342/1605 ... yes 47/1606 ... yes 14/1607 ... yes 391/1608 ... yes 560/1609 ... yes 560/1610 ... yes 549/1611 ... yes 611/1612 ... yes 596/1613 ... yes 398/1614 ... yes 14/1615 ... yes 276/1616 ... yes 439/1617 ... yes 447/1618 ... yes 39/1619 ... yes 14/1620 ... yes 447/1621 ... yes 447/1622 ... yes 39/1623 ... yes 311/1626 ... yes 5/1627 ... yes 5/1628 ... yes 5/1629 ... yes 5/1630 ... yes 5/1631 ... yes 299/1632 ... yes 14/1634 ... yes 5/1635 ... yes 160/1637 ... yes 442/1638 ... yes 351/1639 ... yes 611/1640 ... yes 329/1641 ... yes 6/1642 ... yes 287/1643 ... yes 390/1644 ... yes 299/1645 ... yes 322/1646 ... yes 64/1647 ... yes 628/1648 ... yes 162/1650 ... yes 561/1651 ... yes 439/1652 ... yes 5/1655 ... yes 549/1656 ... yes 568/1657 ... yes 629/1658 ... yes 439/1660 ... yes 649/1661 ... yes 629/1662 ... yes 630/1663 ... yes 331/1665 ... yes 313/1666 ... yes 225/1667 ... yes 340/1668 ... yes 173/1669 ... yes 656/1670 ... yes 634/1671 ... yes 635/1672 ... yes 12/1673 ... yes 24/1674 ... yes 27/1680 ... yes 24/1681 ... yes 472/1682 ... yes 638/1683 ... yes 638/1684 ... yes 638/1685 ... yes 292/1686 ... yes 76/1687 ... yes 397/1688 ... yes 383/1689 ... yes 628/1690 ... yes 6/1691 ... yes 565/1692 ... yes 560/1693 ... yes 525/1694 ... yes 22/1696 ... yes 287/1697 ... yes 406/1698 ... yes 322/1699 ... yes 5/1700 ... yes 40/1701 ... yes 646/1702 ... yes 611/1703 ... yes 181/1706 ... yes 287/1707 ... yes 661/1708 ... yes 391/1709 ... yes 287/1710 ... yes 39/1711 ... yes 644/1712 ... yes 326/1713 ... yes 461/1714 ... yes 246/1716 ... yes 341/1718 ... yes 287/1719 ... yes 341/1720 ... yes 256/1721 ... yes 654/1722 ... yes 654/1723 ... yes 654/1724 ... yes 5/1725 ... yes 173/1726 ... yes 292/1727 ... yes 170/1728 ... yes 655/1729 ... yes 287/1730 ... yes 40/1732 ... yes 658/1733 ... yes 591/1734 ... yes 560/1736 ... yes 645/1737 ... yes 173/1739 ... yes 192/1740 ... yes 440/1742 ... yes 340/1743 ... yes 592/1744 ... yes 560/1745 ... yes 560/1746 ... yes 560/1748 ... yes 560/1749 ... yes 560/1750 ... yes 560/1751 ... yes 286/1752 ... yes 560/1753 ... yes 560/1755 ... yes 560/1756 ... yes 560/1757 ... yes 560/1758 ... yes 560/1759 ... yes 560/1760 ... yes 560/1761 ... yes 560/1762 ... yes 611/1763 ... yes 611/1764 ... yes 565/1765 ... yes 658/1766 ... yes 665/1767 ... yes 665/1768 ... yes 665/1769 ... yes 665/1770 ... yes 665/1771 ... yes 5/1772 ... yes 668/1773 ... yes 592/1774 ... yes 351/1775 ... yes 565/1776 ... yes 655/1777 ... yes 560/1778 ... yes 532/1779 ... yes 292/1780 ... yes 447/1782 ... yes 490/1783 ... yes 628/1785 ... yes 656/1786 ... yes 670/1788 ... yes 351/1789 ... yes 292/1790 ... yes 61/1791 ... yes 540/1793 ... yes 540/1794 ... yes 540/1795 ... yes 540/1796 ... yes 540/1797 ... yes 534/1798 ... yes 540/1799 ... yes 540/1800 ... yes 540/1801 ... yes 540/1802 ... yes 540/1803 ... yes 540/1804 ... yes 540/1805 ... yes 540/1806 ... yes 534/1807 ... yes 540/1808 ... yes 540/1809 ... yes 545/1810 ... yes 540/1811 ... yes 645/1812 ... yes 546/1813 ... yes 540/1814 ... yes 546/1815 ... yes 546/1816 ... yes 545/1817 ... yes 540/1818 ... yes 545/1819 ... yes 546/1820 ... yes 540/1821 ... yes 540/1822 ... yes 545/1823 ... yes 546/1824 ... yes 546/1825 ... yes 545/1826 ... yes 545/1827 ... yes 645/1828 ... yes 540/1829 ... yes 545/1830 ... yes 540/1832 ... yes 545/1833 ... yes 675/1834 ... yes 675/1835 ... yes 674/1836 ... yes 675/1837 ... yes 675/1838 ... yes 675/1839 ... yes 673/1840 ... yes 674/1841 ... yes 674/1842 ... yes 673/1843 ... yes 673/1844 ... yes 673/1845 ... yes 674/1846 ... yes 540/1847 ... yes 673/1848 ... yes 674/1849 ... yes 545/1850 ... yes 546/1851 ... yes 673/1852 ... yes 540/1853 ... yes 545/1854 ... yes 546/1855 ... yes 673/1856 ... yes 673/1857 ... yes 674/1858 ... yes 549/1859 ... yes 674/1860 ... yes 673/1861 ... yes 674/1862 ... yes 673/1863 ... yes 674/1864 ... yes 645/1865 ... yes 634/1866 ... yes 543/1868 ... yes 543/1869 ... yes 543/1870 ... yes 646/1871 ... yes 61/1872 ... yes 491/1873 ... yes 322/1875 ... yes 418/1876 ... yes 358/1877 ... yes 649/1878 ... yes 45/1879 ... yes 634/1880 ... yes 286/1881 ... yes 638/1882 ... yes 507/1883 ... yes 561/1886 ... yes 565/1887 ... yes 596/1888 ... yes 561/1889 ... yes 656/1890 ... yes 532/1891 ... yes 532/1892 ... yes 287/1893 ... yes 24/1894 ... yes 540/1896 ... yes 14/1897 ... yes 192/1898 ... yes 596/1901 ... yes 663/1902 ... yes 225/1903 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.5.3) Git version >= 2.18.0 ? ... yes (2.18.1) Git user has default SSH configuration? ... yes Active users: ... 386 Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished