"Extra Sidekiq processes" setting sidekiq_cluster['queue_groups'] does not take an array
Summary
https://docs.gitlab.com/ee/administration/operations/extra_sidekiq_processes.html
The documentation suggests that should take an array of strings describing queue groups. Only the first element in the array is used. In order to run multiple cluster processes, one must actually use a single string with space-separated queue descriptions.
Fixing this bug might break working installations.
Steps to reproduce
Use:
sidekiq_cluster['queue_groups'] = [ "geo,post_receive,cronjob", "geo,post_receive,cronjob", "geo,post_receive,cronjob" ]
Run gitlab-ctl reconfigure, and the resulting process will be:
ruby /opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster -e production -r /opt/gitlab/embedded/service/gitlab-rails -m 15 geo,post_receive,cronjob --negate
Instead use:
sidekiq_cluster['queue_groups'] = [ "geo,post_receive,cronjob geo,post_receive,cronjob geo,post_receive,cronjob" ]
Run gitlab-ctl reconfigure, and the resulting process will be:
ruby /opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster -e production -r /opt/gitlab/embedded/service/gitlab-rails -m 15 geo,post_receive,cronjob geo,post_receive,cronjob geo,post_receive,cronjob --negate
In the latter case, there are three cluster processes. In the former case, there is only one.
What is the current bug behavior?
Only one cluster will be created by any of the examples given in the documentation, and any additional queue specifications are unused.
What is the expected correct behavior?
One cluster should be created for each array element in the list.
Results of GitLab environment info
Expand for output related to GitLab environment info
[root@dvgitdevvl01 ~]# gitlab-rake gitlab:env:info
System information System: CentOS 7.6.1810 Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.3p62 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 3.2.12 Git Version: 2.22.0 Sidekiq Version:5.2.7 Go Version: unknown
GitLab information Version: 12.4.3-ee Revision: 30671643f41 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.9 URL: https://gitlab.dev.tableausoftware.com HTTP Clone URL: https://gitlab.dev.tableausoftware.com/some-group/some-project.git SSH Clone URL: git@gitlab.dev.tableausoftware.com:some-group/some-project.git Elasticsearch: no Geo: yes Geo node: Primary Using LDAP: yes Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 10.2.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
[root@dvgitdevvl01 ~]# gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 10.2.0 ? ... OK (10.2.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 2 Try fixing it: sudo service gitlab stop sudo pkill -u git -f sidekiq sleep 10 && sudo pkill -9 -u git -f sidekiq sudo service gitlab start Please fix the error above and rerun the checks.
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 100 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 2/1 ... yes 2/2 ... yes 2/3 ... yes 2/4 ... yes 2/5 ... yes 2/6 ... yes 2/7 ... yes 2/8 ... yes 2/9 ... yes 2/10 ... yes 2/11 ... yes 2/13 ... yes 13/14 ... yes 12/15 ... yes 24/16 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.3) Git version >= 2.22.0 ? ... yes (2.22.0) Git user has default SSH configuration? ... yes Active users: ... 23 Is authorized keys file accessible? ... yes Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking Geo ...
GitLab Geo is available ... yes GitLab Geo is enabled ... yes GitLab Geo secondary database is correctly configured ... not a secondary node Database replication enabled? ... not a secondary node Database replication working? ... not a secondary node GitLab Geo tracking database is configured to use Foreign Data Wrapper? ... not a secondary node GitLab Geo tracking database Foreign Data Wrapper schema is up-to-date? ... not a secondary node GitLab Geo HTTP(S) connectivity ... not a secondary node HTTP/HTTPS repository cloning is enabled ... yes Machine clock is synchronized ... yes Git user has default SSH configuration? ... yes OpenSSH configured to use AuthorizedKeysCommand ... skipped Reason: Cannot access OpenSSH configuration file Try fixing it: This is expected if you are using SELinux. You may want to check configuration manually For more information see: doc/administration/operations/fast_ssh_key_lookup.md GitLab configured to disable writing to authorized_keys file ... no Try fixing it: You need to disable
Write to authorized_keys file
in GitLab's Admin panel For more information see: doc/administration/operations/fast_ssh_key_lookup.md GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yesChecking Geo ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)