sidekiq-cluster throws errors for certain gitlab.yml configs
Summary
sidekiq-cluster throws errors if there is blank config in gitlab.yml
that it is referenced by sidekiq-cluster.
sidekiq_1 | {"severity":"INFO","time":"2021-12-22T17:22:28.500Z","message":"Starting cluster with 1 processes"}
sidekiq_1 | /srv/gitlab/sidekiq_cluster/cli.rb:203:in `sidekiq_exporter_enabled?': undefined method `enabled' for nil:NilClass (NoMethodError)
sidekiq_1 | from /srv/gitlab/sidekiq_cluster/cli.rb:231:in `metrics_server_enabled?'
sidekiq_1 | from /srv/gitlab/sidekiq_cluster/cli.rb:191:in `start_metrics_server'
sidekiq_1 | from /srv/gitlab/sidekiq_cluster/cli.rb:106:in `run'
sidekiq_1 | from bin/sidekiq-cluster:11:in `<main>'
Steps to reproduce
- Have a blank
sidekiq_exporter
setting in your gitlab.yml, and run the sidekiq_cluster binary
production: &base
monitoring:
sidekiq_exporter:
What is the current bug behavior?
Sidekiq cluster is missing any of the gitlab initializer logic, so doesn't have any setting defaults in place.
What is the expected correct behavior?
That the gitlab settings work the same in sidekiq_cluster as they do in the rails applicaiton
Possible fixes
Load the settings initializer in sidekiq_cluster
Designs
- Show closed items
- Issue#35247014.81Category:Fuzz Testing GitLab Ultimate Technical Writing UX devops application security testing feature enhancement frontend fuzzing coverage group dynamic analysis section sec tw finished type feature workflow production
- Issue#35246514.81auto updated devops application security testing frontend section sec
- Issue#35207314.8Category:Fuzz Testing GitLab Ultimate backend devops application security testing feature enhancement fuzzing coverage group dynamic analysis section sec type bug workflow verification
- Issue#35163214.8Category:Fuzz Testing GitLab Ultimate devops application security testing feature enhancement frontend fuzzing coverage group dynamic analysis section sec type feature workflow verification
- Issue#35153914.8Category:Fuzz Testing GitLab Ultimate devops application security testing feature enhancement frontend fuzzing coverage group dynamic analysis section sec type feature workflow verification
- Issue#35097514.82Category:Fuzz Testing GitLab Ultimate devops application security testing feature enhancement frontend fuzzing coverage group dynamic analysis section sec type feature workflow verification
- Issue#35097414.82Category:Fuzz Testing GitLab Ultimate devops application security testing feature enhancement frontend fuzzing coverage group dynamic analysis section sec type feature workflow in dev
- Issue#34901414.8Category:Fuzz Testing GitLab Ultimate devops application security testing feature flag frontend fuzzing coverage group dynamic analysis missed:14.7 section sec type feature workflow in dev
- Issue#34748314.7Category:Fuzz Testing GitLab Ultimate devops application security testing feature enhancement frontend fuzzing coverage group dynamic analysis missed:14.6 section sec type bug workflow verification
- Issue#34509014.73Category:Fuzz Testing GitLab Ultimate devops application security testing feature enhancement frontend fuzzing coverage group dynamic analysis section sec type feature workflow verification
- Issue#34508914.73Category:Fuzz Testing auto updated devops application security testing frontend group dynamic analysis section sec workflow verification
- Issue#34508814.63Category:Fuzz Testing GitLab Ultimate devops application security testing feature enhancement frontend fuzzing coverage group dynamic analysis missed:14.5 section sec type feature workflow verification
- Issue#34148614.55Category:Fuzz Testing backend devops application security testing feature addition fuzzing coverage group dynamic analysis section sec type feature
- Issue#34148514.55Category:Fuzz Testing backend devops application security testing feature addition fuzzing coverage group dynamic analysis section sec type feature
- Issue#34148914.82Category:Fuzz Testing Technical Writing backend devops application security testing documentation feature addition fuzzing coverage group dynamic analysis missed:14.6 missed:14.7 section sec tw-weight 8 type feature
- Issue#34148714.65Category:Fuzz Testing backend devops application security testing feature enhancement fuzzing coverage group dynamic analysis section sec type feature
- Issue#34258014.62Category:Fuzz Testing GitLab Ultimate devops application security testing feature enhancement frontend fuzzing coverage group dynamic analysis missed:14.5 section sec type feature workflow verification
- Issue#34258314.82Category:Fuzz Testing GitLab Ultimate devops application security testing feature enhancement frontend fuzzing coverage group dynamic analysis missed:14.6 missed:14.7 section sec type feature workflow verification
- Issue#34243314.63Category:Fuzz Testing GitLab Ultimate devops application security testing feature enhancement frontend fuzzing coverage group dynamic analysis missed:14.5 section sec type feature workflow verification
- Issue#34133814.4automation:ml devops package group package registry section ops
- Issue#34121814.6Category:Package Registry Generic Package Repository auto updated devops package fuzzing coverage group package registry missed:14.4 missed:14.5 section ops
- Epicgitlab-org#668323Dec 18, 2021 – Jan 17, 2022Category:Fuzz Testing backend devops create frontend fuzzing coverage group dynamic analysis missed:14.2 section sec
- Issue#33973214.5Category:Fuzz Testing GitLab Ultimate backend devops application security testing direction fuzzing coverage group dynamic analysis missed:14.3 missed:14.4 section sec type feature workflow in dev
- Epicgitlab-org#640647Jul 18 – Aug 17, 2021Category:Fuzz Testing Deliverable GitLab Ultimate devops application security testing frontend fuzzing coverage group dynamic analysis section sec type feature
- Issue#33494114.7Category:Fuzz Testing Technical Writing backend devops application security testing documentation fuzzing coverage group dynamic analysis missed:14.2 missed:14.3 missed:14.4 missed:14.5 missed:14.6 section sec tw-weight 5 workflow refinement
- Issue#27316914.2Category:Fuzz Testing devops application security testing fuzzing coverage group dynamic analysis missed:13.8 section sec
- Issue#32689513.113Category:Fuzz Testing Deliverable GitLab Ultimate devops application security testing direction frontend fuzzing coverage group dynamic analysis missed-deliverable missed:13.10 section sec type feature workflow in review
- Issue#32689613.112Category:Fuzz Testing Deliverable GitLab Ultimate devops application security testing direction frontend fuzzing coverage group dynamic analysis missed-deliverable missed:13.10 section sec type feature workflow in review
- Issue#32689413.111Category:Fuzz Testing Deliverable GitLab Ultimate devops application security testing direction frontend fuzzing coverage group dynamic analysis missed-deliverable missed:13.10 section sec type feature workflow verification
- Issue#28059213.112Category:Fuzz Testing Deliverable GitLab Ultimate backend devops application security testing direction fuzzing coverage group dynamic analysis section sec type feature workflow in dev
- Issue#32636113.113Category:Fuzz Testing Deliverable GitLab Ultimate backend devops application security testing direction fuzzing coverage group dynamic analysis section sec type feature workflow in dev
- Issue#32636213.112Category:Fuzz Testing Deliverable GitLab Ultimate backend devops application security testing direction fuzzing coverage group dynamic analysis section sec type feature workflow in dev
- Issue#29444413.113Category:Fuzz Testing Deliverable GitLab Ultimate devops application security testing direction frontend fuzzing coverage group dynamic analysis missed-deliverable missed:13.10 section sec type feature workflow verification
- Issue#32161814.3Category:Fuzz Testing backend devops create frontend fuzzing coverage group dynamic analysis missed:14.2 section sec
- Issue#32789714.52Category:Fuzz Testing GitLab Ultimate devops application security testing frontend fuzzing coverage group dynamic analysis section sec type feature workflow production
- Issue#32088613.103Category:Fuzz Testing GitLab Ultimate devops application security testing direction frontend fuzzing coverage group dynamic analysis section sec workflow verification
- Issue#29954413.101Category:Fuzz Testing Deferred UX Deliverable UX devops create frontend group dynamic analysis workflow in dev
- Issue#29442513.92Category:Fuzz Testing GitLab Ultimate backend devops application security testing direction frontend fuzzing coverage group dynamic analysis section sec type feature workflow production
- Issue#28059314.32Category:Fuzz Testing GitLab Ultimate backend devops application security testing direction frontend fuzzing coverage group dynamic analysis section sec type feature workflow production
- Issue#28059013.9Category:Fuzz Testing backend devops application security testing fuzzing coverage group dynamic analysis missed:13.7 section sec type feature workflow ready for development
- Issue#32841814.8Category:Fuzz Testing GitLab Ultimate devops application security testing feature flag frontend fuzzing coverage group dynamic analysis missed:14.7 section sec type feature
- Issue#29074113.10Category:Fuzz Testing Deliverable devops application security testing fuzzing coverage group dynamic analysis missed-deliverable missed:13.8 section sec workflow verification
- Issue#268004Category:Fuzz Testing GitLab Ultimate UX FY21-Q4 backend devops application security testing direction feature enhancement frontend fuzzing coverage group dynamic analysis section sec secure:refinement-backend secure:refinement-frontend type feature workflow refinement
- Issue#32847514.2Category:Fuzz Testing Deliverable GitLab Ultimate devops application security testing frontend fuzzing coverage group dynamic analysis section sec type feature workflow refinement
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- DJ Mountney added devopssystems groupcloud connector typebug labels
added devopssystems groupcloud connector typebug labels
- DJ Mountney changed the description
Compare with previous version changed the description
- Author Contributor
I should also note, that having a blank
sidekiq_exporter:
is actually the default in ourgitlab.yml.example
so source installs are likely to run into this issue.Omnibus and Cloud Native charts explicitly have
enabled
configured by default, so aren't running into the bug at the moment. - DJ Mountney added severity3 label
added severity3 label
- Author Contributor
Workaround is to remove the empty
sidekiq_exporter
line from yourgitlab.yml
- DJ Mountney mentioned in commit gitlab-org/build/CNG@7bc2128c
mentioned in commit gitlab-org/build/CNG@7bc2128c
- DJ Mountney mentioned in merge request gitlab-org/build/CNG!849 (merged)
mentioned in merge request gitlab-org/build/CNG!849 (merged)
- 🤖 GitLab Bot 🤖 added sectioncore platform label
added sectioncore platform label
- Contributor
@mkaeppler @rzwambag do you mind checking this one? Thank you!
Collapse replies - Maintainer
This should be addressed by !77187 (merged)
The problem is that GDK users may decide to not apply diffs to
gitlab.yml
after runninggdk update
, even when they are necessary for the app to function properly. This is unfortunate because it creates incomplete config files. We therefore decided to be more defensive in the application to remedy this.This should not be possible outside of the GDK, since all other environments generate complete config files AFAIK.
1 - Author Contributor
Its also currently broken for source installs, as our default gitab.yml.example has this problem.
- Maintainer
Ah yes good point -- it's too easy to forget about source installs... We did update
gitlab.example.yml
: https://gitlab.com/gitlab-org/gitlab/blob/405c21acc37f5e500344f540a21cfe82b0209e16/config/gitlab.yml.example#L1263However, without manual intervention by the admin, this doesn't solve the issue.
My preference would be to ship the quick fix in !77187 (merged) first, since I'd have to explore how easy it would be to implement #349449. It is easy enough to revert the workaround with a proper fix.
- Maintainer
!77187 (merged) just got merged, that should fix this issue
- Matthias Käppler mentioned in merge request !77187 (merged)
mentioned in merge request !77187 (merged)
- Changzheng Liu closed
closed
- Changzheng Liu changed milestone to %14.7
changed milestone to %14.7
- Changzheng Liu assigned to @rzwambag
assigned to @rzwambag
- Contributor
Kubernetes / Helm chart user here, throwing in his 2 cents.
Came across this issue after disabling prometheus for the sidekiq deployment.
gitlab: sidekiq: metrics: enabled: false
───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── │ File: chart/gitlab/Chart.yaml ───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 1 │ apiVersion: v1 2 │ appVersion: 14.6.1 3 │ description: Web-based Git-repository manager with wiki and issue-tracking features. 4 │ home: https://about.gitlab.com/ 5 │ icon: https://gitlab.com/gitlab-com/gitlab-artwork/raw/master/logo/logo-square.png 6 │ keywords: 7 │ - gitlab 8 │ maintainers: 9 │ - email: support@gitlab.com 10 │ name: GitLab Inc. 11 │ name: gitlab 12 │ sources: 13 │ - https://gitlab.com/gitlab-org/charts/gitlab 14 │ version: 5.6.1 ───────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
bat chart/gitlab/charts/gitlab/charts/sidekiq/Chart.yaml ───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── │ File: chart/gitlab/charts/gitlab/charts/sidekiq/Chart.yaml ───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── 1 │ apiVersion: v1 2 │ appVersion: 14.6.1 3 │ description: Gitlab Sidekiq for asynchronous task processing in rails 4 │ home: https://about.gitlab.com/ 5 │ icon: https://gitlab.com/gitlab-com/gitlab-artwork/raw/master/logo/logo-square.svg 6 │ keywords: 7 │ - gitlab 8 │ - sidekiq 9 │ maintainers: 10 │ - email: support@gitlab.com 11 │ name: GitLab Inc. 12 │ name: sidekiq 13 │ sources: 14 │ - https://gitlab.com/gitlab-org/charts/gitlab/tree/master/charts/gitlab/charts/sidekiq 15 │ - https://gitlab.com/gitlab-org/build/CNG/tree/master/gitlab-sidekiq 16 │ version: 5.6.1 ───────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
After disabling prometheus via
values.yaml
Problem seems to sit here: https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/charts/gitlab/charts/sidekiq/templates/configmap.yaml#L116
... monitoring: ip_whitelist: - 127.0.0.0/8 sidekiq_exporter: {{- if .Values.metrics.enabled }} enabled: true address: 0.0.0.0 port: {{ .Values.metrics.port }} {{- end }} sidekiq_health_checks: {{- if .Values.health_checks.enabled }} enabled: true address: 0.0.0.0 port: {{ .Values.health_checks.port | default .Values.metrics.port }} {{- end }} ...
which produces empty value
sidekiq_exporter
in the configmapsgitlab.yaml.erb
. Manually removing from the configmap (k edit configmap gitlab-sidekiq
) fixed my production system. Collapse replies - Maintainer
Thanks for the heads up! I wasn't aware that Charts were affected, too. !77187 (merged) will go out with 14.7, it should address this problem as well.
- 🤖 GitLab Bot 🤖 added devopsdata stores label and removed devopssystems label
added devopsdata stores label and removed devopssystems label