There was an error running gitlab-ctl reconfigure -- undefined method `[]' for nil:NilClass
### Summary Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb There was an error running gitlab-ctl reconfigure: undefined method ` []' for nil:NilClass ### Steps to reproduce We tried to upgrade gitlab-ce (V.11.0.2 to V.11.8), finally by executing the command “gitlab-ctl reconfigure” we are facing the issue. ### What is the current *bug* behavior? Run command “gitlab-ctl reconfigure” ``` Starting Chef Client, version 13.6.4 resolving cookbooks for run list: ["gitlab"] Synchronizing Cookbooks: - gitlab (0.0.1) - package (0.1.0) - postgresql (0.1.0) - registry (0.1.0) - mattermost (0.1.0) - redis (0.1.0) - consul (0.1.0) - letsencrypt (0.1.0) - nginx (0.1.0) - runit (4.3.0) - gitaly (0.1.0) - acme (3.1.0) - crond (0.1.0) - compat_resource (12.19.1) Installing Cookbook Gems: Compiling Cookbooks... Recipe: gitlab::default * directory[/etc/gitlab] action create (up to date) ================================================================================ Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb ================================================================================ NoMethodError ------------- undefined method `[]' for nil:NilClass Cookbook Trace: --------------- /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/definitions/redis_service.rb:21:in `block in f rom_file' /opt/gitlab/embedded/cookbooks/cache/cookbooks/redis/recipes/enable.rb:19:in `from_file' /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:101:in `block in from_file' /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:99:in `each' /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:99:in `from_file' Relevant File Content: ---------------------- /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/definitions/redis_service.rb: 14: # See the License for the specific language governing permissions and 15: # limitations under the License. 16: # 17: 18: define :redis_service, socket_group: nil do 19: svc = params[:name] 20: 21>> redis_dir = node['gitlab'][svc]['dir'] 22: redis_log_dir = node['gitlab'][svc]['log_directory'] 23: redis_user = AccountHelper.new(node).redis_user 24: omnibus_helper = OmnibusHelper.new(node) 25: 26: account 'user and group for redis' do 27: username redis_user 28: uid node['gitlab'][svc]['uid'] 29: ugid redis_user 30: groupname redis_user System Info: ------------ chef_version=13.6.4 platform=ubuntu platform_version=16.04 ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux] program_name=/opt/gitlab/embedded/bin/chef-client executable=/opt/gitlab/embedded/bin/chef-client Running handlers: There was an error running gitlab-ctl reconfigure: undefined method `[]' for nil:NilClass Running handlers complete Chef Client failed. 0 resources updated in 06 seconds ### Relevant logs Run command “sudo gitlab-ctl tail” ==> /var/log/gitlab/gitlab-monitor/state <== ==> /var/log/gitlab/gitlab-monitor/current <== ==> /var/log/gitlab/nginx/gitlab_access.log <== ==> /var/log/gitlab/nginx/error.log <== ==> /var/log/gitlab/nginx/gitlab_error.log <== ==> /var/log/gitlab/nginx/access.log <== ==> /var/log/gitlab/nginx/current <== ==> /var/log/gitlab/unicorn/state <== ==> /var/log/gitlab/unicorn/unicorn_stdout.log <== ==> /var/log/gitlab/unicorn/unicorn_stderr.log <== ==> /var/log/gitlab/unicorn/current <== 2019-03-11_06:55:46.98317 chpst: fatal: unknown user/group: git: 2019-03-11_06:55:47.98600 failed to start a new unicorn master 2019-03-11_06:55:48.00566 starting new unicorn master 2019-03-11_06:55:48.00632 chpst: fatal: unknown user/group: git: 2019-03-11_06:55:49.01351 failed to start a new unicorn master 2019-03-11_06:55:49.04943 starting new unicorn master 2019-03-11_06:55:49.05183 chpst: fatal: unknown user/group: git: 2019-03-11_06:55:50.05987 failed to start a new unicorn master 2019-03-11_06:55:50.09664 starting new unicorn master 2019-03-11_06:55:50.09890 chpst: fatal: unknown user/group: git: ==> /var/log/gitlab/gitaly/state <== ==> /var/log/gitlab/gitaly/current <== 2019-03-11_06:55:41.98902 time="2019-03-11T06:55:41Z" level=info msg="dialing to target with scheme: \"\"" system=system 2019-03-11_06:55:41.98904 time="2019-03-11T06:55:41Z" level=info msg="ccResolverWrapper: sending new addresses to cc: [{/tmp/gitaly-ruby525529577/socket.1 0 <nil>}]" system=system 2019-03-11_06:55:41.98904 time="2019-03-11T06:55:41Z" level=info msg="ClientConn switching balancer to \"pick_first\"" system=system 2019-03-11_06:55:41.98904 time="2019-03-11T06:55:41Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420537220, CONNECTING" system=system 2019-03-11_06:55:41.98905 time="2019-03-11T06:55:41Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420537220, READY" system=system 2019-03-11_06:55:41.98905 time="2019-03-11T06:55:41Z" level=info msg="dialing to target with scheme: \"\"" system=system 2019-03-11_06:55:41.98905 time="2019-03-11T06:55:41Z" level=info msg="ccResolverWrapper: sending new addresses to cc: [{/tmp/gitaly-ruby525529577/socket.0 0 <nil>}]" system=system 2019-03-11_06:55:41.98905 time="2019-03-11T06:55:41Z" level=info msg="ClientConn switching balancer to \"pick_first\"" system=system 2019-03-11_06:55:41.98906 time="2019-03-11T06:55:41Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420724290, CONNECTING" system=system 2019-03-11_06:55:41.98907 time="2019-03-11T06:55:41Z" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc420724290, READY" system=system ==> /var/log/gitlab/redis/state <== ==> /var/log/gitlab/redis/current <== 2019-03-11_06:40:06.71568 1305:M 11 Mar 12:10:06.715 * 10 changes in 300 seconds. Saving... 2019-03-11_06:40:06.71584 1305:M 11 Mar 12:10:06.715 * Background saving started by pid 2179 2019-03-11_06:40:06.71823 2179:C 11 Mar 12:10:06.718 * DB saved on disk 2019-03-11_06:40:06.71849 2179:C 11 Mar 12:10:06.718 * RDB: 0 MB of memory used by copy-on-write 2019-03-11_06:40:06.81567 1305:M 11 Mar 12:10:06.815 * Background saving terminated with success 2019-03-11_06:50:59.03274 1305:M 11 Mar 12:20:59.032 * 10 changes in 300 seconds. Saving... 2019-03-11_06:50:59.03292 1305:M 11 Mar 12:20:59.032 * Background saving started by pid 21261 2019-03-11_06:50:59.03537 21261:C 11 Mar 12:20:59.035 * DB saved on disk 2019-03-11_06:50:59.03562 21261:C 11 Mar 12:20:59.035 * RDB: 0 MB of memory used by copy-on-write 2019-03-11_06:50:59.13342 1305:M 11 Mar 12:20:59.133 * Background saving terminated with success ==> /var/log/gitlab/prometheus/state <== ==> /var/log/gitlab/prometheus/current <== 2019-03-11_06:55:47.93047 time="2019-03-11T12:25:47+05:30" level=info msg="Host details (Linux 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 DevOpsServer (none))" source="main.go:89" 2019-03-11_06:55:47.93901 time="2019-03-11T12:25:47+05:30" level=info msg="Loading configuration file /var/opt/gitlab/prometheus/prometheus.yml" source="main.go:254" 2019-03-11_06:55:47.94338 time="2019-03-11T12:25:47+05:30" level=info msg="Listening on localhost:9090" source="web.go:341" 2019-03-11_06:55:47.94455 time="2019-03-11T12:25:47+05:30" level=error msg="Error opening memory series storage: found existing files in storage path that do not look like storage files compatible with this version of Prometheus; please delete the files in the storage path or choose a different storage path" source="main.go:192" 2019-03-11_06:55:49.11642 time="2019-03-11T12:25:49+05:30" level=info msg="Starting prometheus (version=1.8.2, branch=master, revision=32e516893728d3f2d9773099d1f088e134465a10)" source="main.go:87" 2019-03-11_06:55:49.11647 time="2019-03-11T12:25:49+05:30" level=info msg="Build context (go=go1.9.2, user=GitLab-Omnibus, date=2018-06-26T19:53:58+00:00)" source="main.go:88" 2019-03-11_06:55:49.11653 time="2019-03-11T12:25:49+05:30" level=info msg="Host details (Linux 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 DevOpsServer (none))" source="main.go:89" 2019-03-11_06:55:49.13496 time="2019-03-11T12:25:49+05:30" level=info msg="Loading configuration file /var/opt/gitlab/prometheus/prometheus.yml" source="main.go:254" 2019-03-11_06:55:49.13509 time="2019-03-11T12:25:49+05:30" level=info msg="Listening on localhost:9090" source="web.go:341" 2019-03-11_06:55:49.13754 time="2019-03-11T12:25:49+05:30" level=error msg="Error opening memory series storage: found existing files in storage path that do not look like storage files compatible with this version of Prometheus; please delete the files in the storage path or choose a different storage path" source="main.go:192" ==> /var/log/gitlab/mailroom/state <== ==> /var/log/gitlab/mailroom/current <== 2019-03-11_06:55:49.67157 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/mail_room-0.9.1/lib/mail_room/mailbox_watcher.rb:57:in `new' 2019-03-11_06:55:49.67158 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/mail_room-0.9.1/lib/mail_room/mailbox_watcher.rb:57:in `connection' 2019-03-11_06:55:49.67161 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/mail_room-0.9.1/lib/mail_room/mailbox_watcher.rb:28:in `run' 2019-03-11_06:55:49.67162 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/mail_room-0.9.1/lib/mail_room/coordinator.rb:19:in `each' 2019-03-11_06:55:49.67162 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/mail_room-0.9.1/lib/mail_room/coordinator.rb:19:in `run' 2019-03-11_06:55:49.67163 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/mail_room-0.9.1/lib/mail_room/cli.rb:52:in `start' 2019-03-11_06:55:49.67164 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/mail_room-0.9.1/bin/mail_room:5:in `<top (required)>' 2019-03-11_06:55:49.67165 /opt/gitlab/embedded/bin/mail_room:23:in `load' 2019-03-11_06:55:49.67165 /opt/gitlab/embedded/bin/mail_room:23:in `<top (required)>' 2019-03-11_06:55:49.68816 Runit: waiting 5 seconds before restarting mail_room ==> /var/log/gitlab/postgresql/state <== ==> /var/log/gitlab/postgresql/current <== ==> /var/log/gitlab/node-exporter/state <== ==> /var/log/gitlab/node-exporter/current <== ==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== ==> /var/log/gitlab/gitlab-rails/sidekiq.log <== 2019-03-11_06:55:45.10170 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command' 2019-03-11_06:55:45.10170 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch' 2019-03-11_06:55:45.10170 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli.rb:27:in `dispatch' 2019-03-11_06:55:45.10170 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start' 2019-03-11_06:55:45.10171 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli.rb:18:in `start' 2019-03-11_06:55:45.10171 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/exe/bundle:30:in `block in <top (required)>' 2019-03-11_06:55:45.10171 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors' 2019-03-11_06:55:45.10171 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/exe/bundle:22:in `<top (required)>' 2019-03-11_06:55:45.10172 /opt/gitlab/embedded/bin/bundle:23:in `load' 2019-03-11_06:55:45.10172 /opt/gitlab/embedded/bin/bundle:23:in `<main>' ==> /var/log/gitlab/gitlab-rails/grpc.log <== ==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== ==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2018-07-02-14-24-36.log <== ==> /var/log/gitlab/gitlab-rails/production_json.log <== ==> /var/log/gitlab/gitlab-rails/api_json.log <== ==> /var/log/gitlab/gitlab-rails/production.log <== Creating scope :references_project. Overwriting existing method MergeRequest.references_project. Creating scope :system. Overwriting existing method Note.system. Creating scope :of_projects. Overwriting existing method MergeRequest.of_projects. Creating scope :join_project. Overwriting existing method MergeRequest.join_project. Creating scope :references_project. Overwriting existing method MergeRequest.references_project. Creating scope :system. Overwriting existing method Note.system. Creating scope :of_projects. Overwriting existing method MergeRequest.of_projects. Creating scope :join_project. Overwriting existing method MergeRequest.join_project. Creating scope :references_project. Overwriting existing method MergeRequest.references_project. Creating scope :system. Overwriting existing method Note.system. ==> /var/log/gitlab/gitlab-rails/application.log <== ==> /var/log/gitlab/gitlab-workhorse/state <== ==> /var/log/gitlab/gitlab-workhorse/current <== ==> /var/log/gitlab/alertmanager/state <== ==> /var/log/gitlab/alertmanager/current <== ==> /var/log/gitlab/sidekiq/state <== ==> /var/log/gitlab/sidekiq/current <== 2019-03-11_06:55:45.10170 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command' 2019-03-11_06:55:45.10170 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch' 2019-03-11_06:55:45.10170 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli.rb:27:in `dispatch' 2019-03-11_06:55:45.10170 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start' 2019-03-11_06:55:45.10171 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli.rb:18:in `start' 2019-03-11_06:55:45.10171 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/exe/bundle:30:in `block in <top (required)>' 2019-03-11_06:55:45.10171 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors' 2019-03-11_06:55:45.10171 /opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/exe/bundle:22:in `<top (required)>' 2019-03-11_06:55:45.10172 /opt/gitlab/embedded/bin/bundle:23:in `load' 2019-03-11_06:55:45.10172 /opt/gitlab/embedded/bin/bundle:23:in `<main>' ==> /var/log/gitlab/postgres-exporter/state <== ==> /var/log/gitlab/postgres-exporter/current <== ==> /var/log/gitlab/logrotate/state <== ==> /var/log/gitlab/logrotate/current <== ==> /var/log/gitlab/redis-exporter/state <== ==> /var/log/gitlab/redis-exporter/current <== ==> /var/log/gitlab/unicorn/current <== 2019-03-11_06:55:51.10244 failed to start a new unicorn master 2019-03-11_06:55:51.11312 starting new unicorn master 2019-03-11_06:55:51.11401 chpst: fatal: unknown user/group: git: ==> /var/log/gitlab/prometheus/current <== 2019-03-11_06:55:50.29819 time="2019-03-11T12:25:50+05:30" level=info msg="Starting prometheus (version=1.8.2, branch=master, revision=32e516893728d3f2d9773099d1f088e134465a10)" source="main.go:87" 2019-03-11_06:55:50.29822 time="2019-03-11T12:25:50+05:30" level=info msg="Build context (go=go1.9.2, user=GitLab-Omnibus, date=2018-06-26T19:53:58+00:00)" source="main.go:88" 2019-03-11_06:55:50.29825 time="2019-03-11T12:25:50+05:30" level=info msg="Host details (Linux 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 DevOpsServer (none))" source="main.go:89" 2019-03-11_06:55:50.30992 time="2019-03-11T12:25:50+05:30" level=info msg="Loading configuration file /var/opt/gitlab/prometheus/prometheus.yml" source="main.go:254" 2019-03-11_06:55:50.31004 time="2019-03-11T12:25:50+05:30" level=info msg="Listening on localhost:9090" source="web.go:341" 2019-03-11_06:55:50.31233 time="2019-03-11T12:25:50+05:30" level=error msg="Error opening memory series storage: found existing files in storage path that do not look like storage files compatible with this version of Prometheus; please delete the files in the storage path or choose a different storage path" source="main.go:192" ==> /var/log/gitlab/unicorn/current <== 2019-03-11_06:55:52.11851 failed to start a new unicorn master 2019-03-11_06:55:52.14863 starting new unicorn master 2019-03-11_06:55:52.15109 chpst: fatal: unknown user/group: git: ==> /var/log/gitlab/prometheus/current <== 2019-03-11_06:55:51.37671 time="2019-03-11T12:25:51+05:30" level=info msg="Starting prometheus (version=1.8.2, branch=master, revision=32e516893728d3f2d9773099d1f088e134465a10)" source="main.go:87" 2019-03-11_06:55:51.37674 time="2019-03-11T12:25:51+05:30" level=info msg="Build context (go=go1.9.2, user=GitLab-Omnibus, date=2018-06-26T19:53:58+00:00)" source="main.go:88" 2019-03-11_06:55:51.37674 time="2019-03-11T12:25:51+05:30" level=info msg="Host details (Linux 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 DevOpsServer (none))" source="main.go:89" 2019-03-11_06:55:51.38034 time="2019-03-11T12:25:51+05:30" level=info msg="Loading configuration file /var/opt/gitlab/prometheus/prometheus.yml" source="main.go:254" 2019-03-11_06:55:51.38090 time="2019-03-11T12:25:51+05:30" level=info msg="Listening on localhost:9090" source="web.go:341" 2019-03-11_06:55:51.38221 time="2019-03-11T12:25:51+05:30" level=error msg="Error opening memory series storage: found existing files in storage path that do not look like storage files compatible with this version of Prometheus; please delete the files in the storage path or choose a different storage path" source="main.go:192" ``` ### Details of package version <details> ``` Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description ii gitlab-ce 11.8.1-ce.0 amd64 GitLab Community Edition (including NGINX, Postgres, Redis) ``` </details> ### Environment details * Operating System: `Ubuntu 16.04` * Installation Target, remove incorrect values: * Bare Metal Machine * Installation Type, remove incorrect values: * Upgrade from version ` gitlab-ce (V.11.0.2 to V.11.8)` * Is there any other software running on the machine: ` NGINX, Postgres, Redis, ` * Is this a single or multiple node installation? `Single` * Resources * CPU: ` cpu family : 6 model : 62 model name : Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz stepping : 4 ` * Memory total: ` MemTotal: 6110480 kB MemFree: 989044 kB MemAvailable: 4351760 kB ` ### Configuration details <details> ``` external_url 'http://*******' gitlab_rails['gitlab_email_enabled'] = true gitlab_rails['gitlab_email_from'] = 'krishnakvs@*****.com' gitlab_rails['gitlab_email_display_name'] = 'Tech' gitlab_rails['gitlab_email_reply_to'] = 'noreply@example.com' gitlab_rails['gitlab_default_can_create_group'] = true gitlab_rails['gitlab_username_changing_enabled'] = true gitlab_rails['gitlab_default_theme'] = 2 gitlab_rails['gitlab_default_projects_features_issues'] = true gitlab_rails['gitlab_default_projects_features_merge_requests'] = true gitlab_rails['gitlab_default_projects_features_wiki'] = true gitlab_rails['gitlab_default_projects_features_snippets'] = true gitlab_rails['gitlab_default_projects_features_builds'] = true gitlab_rails['gitlab_default_projects_features_container_registry'] = true gitlab_rails['webhook_timeout'] = 25 gitlab_rails['incoming_email_enabled'] = true gitlab_rails['incoming_email_address'] = "gitlab-incoming+%{key}Krishnakvs@*****.com" gitlab_rails['incoming_email_email'] = "Krishnakvs@*****.com" gitlab_rails['incoming_email_password'] = "********" gitlab_rails['incoming_email_host'] = "imap.gmail.com" gitlab_rails['incoming_email_port'] = 993 gitlab_rails['incoming_email_ssl'] = true gitlab_rails['incoming_email_start_tls'] = false gitlab_rails['incoming_email_mailbox_name'] = "inbox" gitlab_rails['incoming_email_idle_timeout'] = 60 gitlab_rails['ldap_enabled'] = true gitlab_rails['ldap_servers'] = YAML.load <<-'EOS' main: # 'main' is the GitLab 'provider ID' of this LDAP server label: 'LDAP' host: '*****.LOCAL' port: 389 uid: 'sAMAccountName' bind_dn: 'CN=****,OU= Admin User,DC=****,DC=LOCAL' password: '*****' method: 'plain' # "start_tls" or "simple_tls" or "plain" verify_certificates: true active_directory: true allow_username_or_email_login: false lowercase_usernames: false block_auto_created_users: false base: 'CN=gitlab,OU=gitlab,DC=TECHNO,DC=LOCAL' EOS gitlab_rails['backup_path'] = "/mnt/gitlab-backup" ``` </details>
issue