Sidekiq keeps restarting after upgrade to 17.0.0
<!---
Please read this!
Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "type::bug" label:
- https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression
- https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=type::bug
and verify the issue you're about to submit isn't a duplicate.
--->
### Summary
After updating GitLab to 17.0.0 Sidekiq keeps restarting over and over again.
### Steps to reproduce
Update GitLab to version 17.0.0
### What is the current *bug* behavior?
Sidekiq crashes over and over again and normal operation of GitLab seems to be no possible (No pipelines start running, background migrations stuck)
### What is the expected *correct* behavior?
No subsequent crashes of Sidekiq
### Relevant logs and/or screenshots
Example entries from `/var/log/gitlab/sidekiq/current`:
```
/opt/gitlab/embedded/service/gitlab-rails/sidekiq_cluster/cli.rb:62:in `run': invalid option: -m (OptionParser::InvalidOption)
from /opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster:11:in `<main>'
/opt/gitlab/embedded/service/gitlab-rails/sidekiq_cluster/cli.rb:62:in `run': invalid option: m (OptionParser::InvalidOption)
from /opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster:11:in `<main>'
```
### Output of checks
<!-- If you are reporting a bug on GitLab.com, uncomment below -->
<!-- This bug happens on GitLab.com -->
<!-- /label ~"reproduced on GitLab.com" -->
#### Results of GitLab environment info
<!-- Input any relevant GitLab environment information if needed. -->
<details>
<summary>Expand for output related to GitLab environment info</summary>
<pre>
System information
System: Debian 11
Current User: git
Using RVM: no
Ruby Version: 3.1.5p253
Gem Version: 3.5.9
Bundler Version:2.5.9
Rake Version: 13.0.6
Redis Version: 7.0.15
Sidekiq Version:7.1.6
Go Version: unknown
GitLab information
Version: 17.0.0
Revision: d29650c9a04
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 14.11
URL: http://gitlabserver
HTTP Clone URL: http://gitlabserver/some-group/some-project.git
SSH Clone URL: git@gitlabserver:some-group/some-project.git
Using LDAP: yes
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 14.35.0
Repository storages:
- default: unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Gitaly
- default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version: 17.0.0
- default Git Version: 2.44.0.gl2
</pre>
</details>
#### Results of GitLab application Check
<!-- Input any relevant GitLab application check information if needed. -->
<details>
<summary>Expand for output related to the GitLab application check</summary>
<pre>
System information
System: Debian 11
Current User: git
Using RVM: no
Ruby Version: 3.1.5p253
Gem Version: 3.5.9
Bundler Version:2.5.9
Rake Version: 13.0.6
Redis Version: 7.0.15
Sidekiq Version:7.1.6
Go Version: unknown
GitLab information
Version: 17.0.0
Revision: d29650c9a04
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 14.11
URL: http://gitlabserver
HTTP Clone URL: http://gitlabserver/some-group/some-project.git
SSH Clone URL: git@gitlabserver:some-group/some-project.git
Using LDAP: yes
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 14.35.0
Repository storages:
- default: unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Gitaly
- default Address: unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version: 17.0.0
- default Git Version: 2.44.0.gl2
root@gitlabserver:~# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.35.0 ? ... OK (14.35.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... no
Try fixing it:
sudo -u git -H RAILS_ENV=production bin/background_jobs start
For more information see:
doc/install/installation.md in section "Install Init Script"
see log/sidekiq.log for possible errors
Please fix the error above and rerun the checks.
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain
not verifying SSL hostname of LDAPS server 'COMEDDC2016.comed.local:636'
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 92 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
Tables are truncated? ... skipped
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
2/1 ... yes
8/2 ... yes
2/3 ... yes
1/4 ... yes
78/7 ... yes
8/9 ... yes
8/10 ... yes
8/11 ... yes
8/12 ... yes
8/13 ... yes
8/14 ... yes
8/15 ... yes
9/16 ... yes
2/17 ... yes
21/23 ... yes
8/24 ... yes
8/25 ... yes
8/26 ... yes
8/27 ... yes
8/28 ... yes
8/29 ... yes
8/30 ... yes
8/31 ... yes
8/33 ... yes
29/36 ... yes
8/37 ... yes
29/38 ... yes
29/39 ... yes
29/40 ... yes
29/41 ... yes
29/42 ... yes
8/44 ... yes
22/45 ... yes
8/46 ... yes
8/47 ... yes
8/48 ... yes
35/49 ... yes
35/51 ... yes
35/52 ... yes
21/54 ... yes
21/56 ... yes
21/57 ... yes
8/58 ... yes
8/59 ... yes
38/60 ... yes
34/62 ... yes
8/63 ... yes
21/64 ... yes
8/66 ... yes
8/68 ... yes
21/70 ... yes
21/71 ... yes
21/72 ... yes
21/73 ... yes
35/74 ... yes
21/75 ... yes
21/77 ... yes
21/78 ... yes
8/79 ... yes
35/80 ... yes
21/82 ... yes
47/83 ... yes
78/84 ... yes
8/86 ... yes
34/88 ... yes
34/89 ... yes
8/91 ... yes
21/92 ... yes
21/94 ... yes
8/95 ... yes
8/96 ... yes
8/97 ... yes
35/99 ... yes
8/100 ... yes
35/101 ... yes
29/102 ... yes
29/103 ... yes
2/104 ... yes
58/109 ... yes
58/110 ... yes
58/111 ... yes
58/112 ... yes
58/113 ... yes
58/114 ... yes
58/115 ... yes
58/116 ... yes
58/117 ... yes
33/119 ... yes
2/120 ... yes
61/122 ... yes
21/123 ... yes
65/127 ... yes
2/128 ... yes
69/130 ... yes
2/132 ... yes
1/134 ... yes
69/135 ... yes
69/136 ... yes
72/141 ... yes
72/142 ... yes
72/143 ... yes
69/144 ... yes
69/145 ... yes
8/146 ... yes
76/148 ... yes
76/149 ... yes
76/151 ... yes
21/152 ... yes
76/153 ... yes
10/155 ... yes
76/156 ... yes
76/158 ... yes
8/159 ... yes
78/160 ... yes
79/161 ... yes
80/164 ... yes
100/165 ... yes
69/166 ... yes
8/167 ... yes
35/168 ... yes
88/170 ... yes
33/171 ... yes
76/172 ... yes
8/173 ... yes
29/174 ... yes
76/175 ... yes
80/176 ... yes
80/177 ... yes
80/178 ... yes
45/179 ... yes
80/181 ... yes
35/182 ... yes
91/183 ... yes
38/184 ... yes
76/185 ... yes
21/186 ... yes
76/187 ... yes
98/188 ... yes
100/189 ... yes
103/190 ... yes
103/191 ... yes
103/192 ... yes
8/193 ... yes
8/194 ... yes
76/195 ... yes
103/196 ... yes
61/197 ... yes
2/198 ... yes
35/199 ... yes
107/202 ... yes
101/203 ... yes
101/204 ... yes
45/205 ... yes
21/206 ... yes
45/207 ... yes
35/208 ... yes
100/210 ... yes
76/212 ... yes
78/213 ... yes
78/214 ... yes
110/215 ... yes
112/216 ... yes
113/217 ... yes
103/218 ... yes
78/219 ... yes
8/221 ... yes
102/224 ... yes
14/227 ... yes
102/228 ... yes
35/229 ... yes
103/230 ... yes
103/231 ... yes
8/232 ... yes
8/233 ... yes
8/234 ... yes
8/235 ... yes
8/236 ... yes
8/237 ... yes
8/238 ... yes
8/239 ... yes
8/240 ... yes
8/241 ... yes
8/242 ... yes
8/243 ... yes
103/244 ... yes
323/245 ... yes
101/247 ... yes
8/248 ... yes
71/249 ... yes
335/250 ... yes
35/251 ... yes
330/252 ... yes
38/253 ... yes
330/254 ... yes
330/255 ... yes
342/256 ... yes
380/258 ... yes
380/259 ... yes
102/260 ... yes
380/261 ... yes
330/262 ... yes
8/264 ... yes
380/265 ... yes
8/268 ... yes
359/269 ... yes
359/270 ... yes
359/271 ... yes
380/273 ... yes
8/275 ... yes
102/277 ... yes
102/280 ... yes
373/281 ... yes
373/282 ... yes
102/283 ... yes
100/284 ... yes
342/285 ... yes
342/286 ... yes
342/287 ... yes
342/288 ... yes
101/289 ... yes
101/290 ... yes
342/291 ... yes
342/292 ... yes
342/293 ... yes
8/294 ... yes
373/295 ... yes
69/296 ... yes
8/297 ... yes
8/298 ... yes
8/299 ... yes
8/300 ... yes
101/301 ... yes
35/302 ... yes
107/304 ... yes
103/305 ... yes
8/306 ... yes
373/307 ... yes
366/308 ... yes
11/309 ... yes
8/310 ... yes
8/311 ... yes
418/312 ... yes
38/313 ... yes
421/314 ... yes
421/315 ... yes
335/316 ... yes
9/317 ... yes
79/318 ... yes
8/319 ... yes
8/320 ... yes
8/321 ... yes
8/322 ... yes
8/323 ... yes
Redis version >= 6.2.14? ... yes
Ruby version >= 3.0.6 ? ... yes (3.1.5)
Git user has default SSH configuration? ... yes
Active users: ... 55
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
</pre>
</details>
issue