Sidekiq keeps restarting after upgrade to 17.0.0
Summary
After updating GitLab to 17.0.0 Sidekiq keeps restarting over and over again.
Steps to reproduce
Update GitLab to version 17.0.0
What is the current bug behavior?
Sidekiq crashes over and over again and normal operation of GitLab seems to be no possible (No pipelines start running, background migrations stuck)
What is the expected correct behavior?
No subsequent crashes of Sidekiq
Relevant logs and/or screenshots
Example entries from /var/log/gitlab/sidekiq/current
:
/opt/gitlab/embedded/service/gitlab-rails/sidekiq_cluster/cli.rb:62:in `run': invalid option: -m (OptionParser::InvalidOption)
from /opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster:11:in `<main>'
/opt/gitlab/embedded/service/gitlab-rails/sidekiq_cluster/cli.rb:62:in `run': invalid option: m (OptionParser::InvalidOption)
from /opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster:11:in `<main>'
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Debian 11 Current User: git Using RVM: no Ruby Version: 3.1.5p253 Gem Version: 3.5.9 Bundler Version:2.5.9 Rake Version: 13.0.6 Redis Version: 7.0.15 Sidekiq Version:7.1.6 Go Version: unknown GitLab information Version: 17.0.0 Revision: d29650c9a04 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 14.11 URL: http://gitlabserver HTTP Clone URL: http://gitlabserver/some-group/some-project.git SSH Clone URL: git@gitlabserver:some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 14.35.0 Repository storages: - default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Gitaly - default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket - default Version: 17.0.0 - default Git Version: 2.44.0.gl2
Results of GitLab application Check
Expand for output related to the GitLab application check
System information System: Debian 11 Current User: git Using RVM: no Ruby Version: 3.1.5p253 Gem Version: 3.5.9 Bundler Version:2.5.9 Rake Version: 13.0.6 Redis Version: 7.0.15 Sidekiq Version:7.1.6 Go Version: unknown
GitLab information Version: 17.0.0 Revision: d29650c9a04 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 14.11 URL: http://gitlabserver HTTP Clone URL: http://gitlabserver/some-group/some-project.git SSH Clone URL: git@gitlabserver:some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 14.35.0 Repository storages:
- default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Gitaly
- default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version: 17.0.0
- default Git Version: 2.44.0.gl2 root@gitlabserver:~# gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.35.0 ? ... OK (14.35.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... no Try fixing it: sudo -u git -H RAILS_ENV=production bin/background_jobs start For more information see: doc/install/installation.md in section "Install Init Script" see log/sidekiq.log for possible errors Please fix the error above and rerun the checks.
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain not verifying SSL hostname of LDAPS server 'COMEDDC2016.comed.local:636' LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 92 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes Tables are truncated? ... skipped All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Cable config exists? ... yes Resque config exists? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 2/1 ... yes 8/2 ... yes 2/3 ... yes 1/4 ... yes 78/7 ... yes 8/9 ... yes 8/10 ... yes 8/11 ... yes 8/12 ... yes 8/13 ... yes 8/14 ... yes 8/15 ... yes 9/16 ... yes 2/17 ... yes 21/23 ... yes 8/24 ... yes 8/25 ... yes 8/26 ... yes 8/27 ... yes 8/28 ... yes 8/29 ... yes 8/30 ... yes 8/31 ... yes 8/33 ... yes 29/36 ... yes 8/37 ... yes 29/38 ... yes 29/39 ... yes 29/40 ... yes 29/41 ... yes 29/42 ... yes 8/44 ... yes 22/45 ... yes 8/46 ... yes 8/47 ... yes 8/48 ... yes 35/49 ... yes 35/51 ... yes 35/52 ... yes 21/54 ... yes 21/56 ... yes 21/57 ... yes 8/58 ... yes 8/59 ... yes 38/60 ... yes 34/62 ... yes 8/63 ... yes 21/64 ... yes 8/66 ... yes 8/68 ... yes 21/70 ... yes 21/71 ... yes 21/72 ... yes 21/73 ... yes 35/74 ... yes 21/75 ... yes 21/77 ... yes 21/78 ... yes 8/79 ... yes 35/80 ... yes 21/82 ... yes 47/83 ... yes 78/84 ... yes 8/86 ... yes 34/88 ... yes 34/89 ... yes 8/91 ... yes 21/92 ... yes 21/94 ... yes 8/95 ... yes 8/96 ... yes 8/97 ... yes 35/99 ... yes 8/100 ... yes 35/101 ... yes 29/102 ... yes 29/103 ... yes 2/104 ... yes 58/109 ... yes 58/110 ... yes 58/111 ... yes 58/112 ... yes 58/113 ... yes 58/114 ... yes 58/115 ... yes 58/116 ... yes 58/117 ... yes 33/119 ... yes 2/120 ... yes 61/122 ... yes 21/123 ... yes 65/127 ... yes 2/128 ... yes 69/130 ... yes 2/132 ... yes 1/134 ... yes 69/135 ... yes 69/136 ... yes 72/141 ... yes 72/142 ... yes 72/143 ... yes 69/144 ... yes 69/145 ... yes 8/146 ... yes 76/148 ... yes 76/149 ... yes 76/151 ... yes 21/152 ... yes 76/153 ... yes 10/155 ... yes 76/156 ... yes 76/158 ... yes 8/159 ... yes 78/160 ... yes 79/161 ... yes 80/164 ... yes 100/165 ... yes 69/166 ... yes 8/167 ... yes 35/168 ... yes 88/170 ... yes 33/171 ... yes 76/172 ... yes 8/173 ... yes 29/174 ... yes 76/175 ... yes 80/176 ... yes 80/177 ... yes 80/178 ... yes 45/179 ... yes 80/181 ... yes 35/182 ... yes 91/183 ... yes 38/184 ... yes 76/185 ... yes 21/186 ... yes 76/187 ... yes 98/188 ... yes 100/189 ... yes 103/190 ... yes 103/191 ... yes 103/192 ... yes 8/193 ... yes 8/194 ... yes 76/195 ... yes 103/196 ... yes 61/197 ... yes 2/198 ... yes 35/199 ... yes 107/202 ... yes 101/203 ... yes 101/204 ... yes 45/205 ... yes 21/206 ... yes 45/207 ... yes 35/208 ... yes 100/210 ... yes 76/212 ... yes 78/213 ... yes 78/214 ... yes 110/215 ... yes 112/216 ... yes 113/217 ... yes 103/218 ... yes 78/219 ... yes 8/221 ... yes 102/224 ... yes 14/227 ... yes 102/228 ... yes 35/229 ... yes 103/230 ... yes 103/231 ... yes 8/232 ... yes 8/233 ... yes 8/234 ... yes 8/235 ... yes 8/236 ... yes 8/237 ... yes 8/238 ... yes 8/239 ... yes 8/240 ... yes 8/241 ... yes 8/242 ... yes 8/243 ... yes 103/244 ... yes 323/245 ... yes 101/247 ... yes 8/248 ... yes 71/249 ... yes 335/250 ... yes 35/251 ... yes 330/252 ... yes 38/253 ... yes 330/254 ... yes 330/255 ... yes 342/256 ... yes 380/258 ... yes 380/259 ... yes 102/260 ... yes 380/261 ... yes 330/262 ... yes 8/264 ... yes 380/265 ... yes 8/268 ... yes 359/269 ... yes 359/270 ... yes 359/271 ... yes 380/273 ... yes 8/275 ... yes 102/277 ... yes 102/280 ... yes 373/281 ... yes 373/282 ... yes 102/283 ... yes 100/284 ... yes 342/285 ... yes 342/286 ... yes 342/287 ... yes 342/288 ... yes 101/289 ... yes 101/290 ... yes 342/291 ... yes 342/292 ... yes 342/293 ... yes 8/294 ... yes 373/295 ... yes 69/296 ... yes 8/297 ... yes 8/298 ... yes 8/299 ... yes 8/300 ... yes 101/301 ... yes 35/302 ... yes 107/304 ... yes 103/305 ... yes 8/306 ... yes 373/307 ... yes 366/308 ... yes 11/309 ... yes 8/310 ... yes 8/311 ... yes 418/312 ... yes 38/313 ... yes 421/314 ... yes 421/315 ... yes 335/316 ... yes 9/317 ... yes 79/318 ... yes 8/319 ... yes 8/320 ... yes 8/321 ... yes 8/322 ... yes 8/323 ... yes Redis version >= 6.2.14? ... yes Ruby version >= 3.0.6 ? ... yes (3.1.5) Git user has default SSH configuration? ... yes Active users: ... 55 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished