gitlab-ctl reconfigure fails on redis node in HA Geo setup
Summary
During the upgrade of an HA Geo cluster we observed an issue on the redis node that prevented gitlab-ctl reconfigure
to complete after running apt-get install gitlab-ee=12.0.9-ee.0
. Rerunning gitlab-ctl reconfigure
finishes successfully
Steps to reproduce
- Follow instructions described in !3577 (merged) until section "
**On all other nodes (not the deploy node)**
" in the Primary upgrade section (same also occurs on the Secondary) - Upgrade package on redis node (both on primary and secondary)
- Run
sudo gitlab-ctl reconfigure
What is the current bug behavior?
Reconfigure fails
What is the expected correct behavior?
Reconfigure succeeds.
Relevant logs
Relevant logs
Running handlers: There was an error running gitlab-ctl reconfigure: execute[clear the gitlab-rails cache] (gitlab::gitlab-rails line 400) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1' ---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ---- STDOUT: STDERR: rake aborted! Redis::CommandError: NOAUTH Authentication required. /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:15:in `block (6 levels) in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/lib/tasksRecipe: gitlab::gitlab-rails * execute[clear the gitlab-rails cache] action run [execute] rake aborted! Redis::CommandError: NOAUTH Authentication required. /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:15:in `block (6 levels) in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:14:in `loop' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:14:in `block (5 levels) in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:12:in `each' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:12:in `block (4 levels) in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/redis/wrapper.rb:19:in `block in with' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/redis/wrapper.rb:19:in `with' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:8:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in `load' /opt/gitlab/embedded/bin/bundle:23:in `<main>' Tasks: TOP => cache:clear => cache:clear:redis (See full trace by running task with --trace) ================================================================================ Error executing action `run` on resource 'execute[clear the gitlab-rails cache]' ================================================================================ Mixlib::ShellOut::ShellCommandFailed ------------------------------------ Expected process to exit with [0], but received '1' ---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ---- STDOUT: STDERR: rake aborted! Redis::CommandError: NOAUTH Authentication required. /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:15:in `block (6 levels) in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:14:in `loop' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:14:in `block (5 levels) in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:12:in `each' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:12:in `block (4 levels) in <top (required)>' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/redis/wrapper.rb:19:in `block in with' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/redis/wrapper.rb:19:in `with' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:8:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in `load' /opt/gitlab/embedded/bin/bundle:23:in `<main>' Tasks: TOP => cache:clear => cache:clear:redis (See full trace by running task with --trace) ---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ---- Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1 Resource Declaration: --------------------- # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb 400: execute "clear the gitlab-rails cache" do 401: command "/opt/gitlab/bin/gitlab-rake cache:clear" 402: action :nothing 403: not_if { omnibus_helper.not_listening?('redis') || !node['gitlab']['gitlab-rails']['rake_cache_clear'] } 404: end 405: Compiled Resource: ------------------ # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb:400:in `from_file' execute("clear the gitlab-rails cache") do action [:nothing] default_guard_interpreter :execute command "/opt/gitlab/bin/gitlab-rake cache:clear" backup 5 declared_type :execute cookbook_name "gitlab" recipe_name "gitlab-rails" user nil domain nil not_if { #code block } end System Info: ------------ chef_version=14.13.11 platform=ubuntu platform_version=18.04 ruby=ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux] program_name=/opt/gitlab/embedded/bin/chef-client executable=/opt/gitlab/embedded/bin/chef-client
Details of package version
Provide the package version installation details
PRIMARY HA APP01external_url 'http://primary-ha.gogitlab.ml' roles ['application_role'] nginx['enable'] = true gitaly['enable'] = false git_data_dirs({ 'default' => { 'path' => '/var/opt/gitlab/git-data', 'gitaly_address' => 'tcp://10.142.0.11 :8075' } }) gitlab_rails['gitaly_token'] = '' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' gitlab_rails['db_host'] = '10.142.0.9' # IP/hostname of database server gitlab_rails['db_password'] = '' gitlab_rails['redis_port'] = '6379' gitlab_rails['redis_host'] = '10.142.0.10' # IP/hostname of Redis server gitlab_rails['redis_password'] = '' roles ['geo_primary_role']
PRIMARY HA APP02
external_url 'http://primary-ha.gogitlab.ml' roles ['application_role'] nginx['enable'] = true gitaly['enable'] = false git_data_dirs({ 'default' => { 'path' => '/var/opt/gitlab/git-data', 'gitaly_address' => 'tcp://10.142.0.11 :8075' } }) gitlab_rails['gitaly_token'] = '' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' gitlab_rails['db_host'] = '10.142.0.9' # IP/hostname of database server gitlab_rails['db_password'] = '' gitlab_rails['redis_port'] = '6379' gitlab_rails['redis_host'] = '10.142.0.10' # IP/hostname of Redis server gitlab_rails['redis_password'] = '' roles ['geo_primary_role'] gitlab_rails['auto_migrate'] = false
PRIMARY HA GITALY
external_url 'http://gitlab.example.com' postgresql['enable'] = false redis['enable'] = false nginx['enable'] = false prometheus['enable'] = false unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false gitlab_rails['rake_cache_clear'] = false gitlab_rails['auto_migrate'] = false gitlab_rails['internal_api_url'] = 'http://primary-ha.gogitlab.ml' gitaly['listen_addr'] = "10.142.0.11:8075" gitaly['auth_token'] = '' gitaly['storage'] = [ { 'name' => 'default', 'path' => '/var/opt/gitlab/git-data' }, ]
PRIMARY HA PoSTGRESQL
external_url 'http://primary-ha-postgresql.gogitlab.ml' roles ['geo_primary_role', 'postgres_role'] repmgr['enable'] = false consul['enable'] = false prometheus['enable'] = false alertmanager['enable'] = false pgbouncer_exporter['enable'] = false redis_exporter['enable'] = false gitlab_monitor['enable'] = false postgresql['listen_address'] = '10.142.0.9' postgresql['port'] = 5432 postgresql['sql_user_password'] = '' postgresql['trust_auth_cidr_addresses'] = %w(10.142.0.12/32 10.142.0.13/32) postgresql['md5_auth_cidr_addresses'] = ['10.142.0.9/32', '10.164.0.30/32'] gitlab_rails['auto_migrate'] = false postgresql['max_replication_slots'] = 1
PRIMARY HA REDIS
external_url 'http://primary-ha-redis.gogitlab.ml' redis['enable'] = true sidekiq['enable'] = false gitlab_workhorse['enable'] = false unicorn['enable'] = false postgresql['enable'] = false nginx['enable'] = false prometheus['enable'] = false alertmanager['enable'] = false pgbouncer_exporter['enable'] = false gitlab_monitor['enable'] = false gitaly['enable'] = false redis['bind'] = '10.142.0.10' redis['port'] = 6379 redis['password'] = 'password' gitlab_rails['auto_migrate'] = false
SECONDARY HA APP01
external_url 'http://secondary-ha.gogitlab.ml' roles ['geo_secondary_role', 'application_role'] nginx['enable'] = true gitlab_rails['auto_migrate'] = false geo_secondary['db_host'] = '10.164.0.31' geo_secondary['db_password'] = '' geo_postgresql['enable'] = false gitlab_rails['db_host'] = '10.164.0.30' gitlab_rails['db_password'] = '' gitlab_rails['redis_host'] = '10.164.0.32' gitlab_rails['redis_password'] = '' gitaly['enable'] = false git_data_dirs({ 'default' => { 'path' => '/var/opt/gitlab/git-data', 'gitaly_address' => 'tcp://10.164.0.33 :8075' } }) gitlab_rails['gitaly_token'] = ''
SECONDARY HA APP02
external_url 'http://secondary-ha.gogitlab.ml' roles ['geo_secondary_role', 'application_role'] gitlab_rails['auto_migrate'] = false nginx['enable'] = true geo_secondary['db_host'] = '10.164.0.31' geo_secondary['db_password'] = '' geo_postgresql['enable'] = false gitlab_rails['db_host'] = '10.164.0.30' gitlab_rails['db_password'] = '' gitlab_rails['redis_host'] = '10.164.0.32' gitlab_rails['redis_password'] = '' gitaly['enable'] = false git_data_dirs({ 'default' => { 'path' => '/var/opt/gitlab/git-data', 'gitaly_address' => 'tcp://10.164.0.33 :8075' } }) gitlab_rails['gitaly_token'] = '' geo_secondary['auto_migrate'] = false
SECONDARY HA GITALY
external_url 'http://gitlab.example.com' postgresql['enable'] = false redis['enable'] = false nginx['enable'] = false prometheus['enable'] = false unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false gitlab_rails['rake_cache_clear'] = false gitlab_rails['auto_migrate'] = false gitlab_rails['internal_api_url'] = 'http://secondary-ha.gogitlab.ml' gitaly['listen_addr'] = "10.164.0.33:8075" gitaly['auth_token'] = '' gitaly['storage'] = [ { 'name' => 'default', 'path' => '/var/opt/gitlab/git-data' }, ] geo_secondary['auto_migrate'] = false
SECONDARY HA POSTGRESQL
external_url 'http://secondary-ha-postgresql.gogitlab.ml' roles ['postgres_role'] repmgr['enable'] = false postgresql['listen_address'] = '10.164.0.30' postgresql['md5_auth_cidr_addresses'] = ['10.164.0.30/32', '10.164.0.31/32', '10.164.0.34/32' , '10.164.0.35/32'] postgresql['hot_standby'] = 'on' postgresql['sql_user_password'] = '' gitlab_rails['db_password'] = '' geo_postgresql['enable'] = false gitlab_rails['auto_migrate'] = false geo_secondary['auto_migrate'] = false
SECONDARY HA REDIS
external_url 'http://secondary-ha-redis.gogitlab.ml' redis['enable'] = true sidekiq['enable'] = false gitlab_workhorse['enable'] = false unicorn['enable'] = false postgresql['enable'] = false nginx['enable'] = false prometheus['enable'] = false alertmanager['enable'] = false pgbouncer_exporter['enable'] = false gitlab_monitor['enable'] = false gitaly['enable'] = false redis['bind'] = '10.164.0.32' redis['port'] = 6379 redis['password'] = '' gitlab_rails['auto_migrate'] = false geo_secondary['auto_migrate'] = false
SECONDARY HA TRACKING
external_url 'http://secondary-ha-tracking.gogitlab.ml' geo_postgresql['enable'] = true geo_postgresql['listen_address'] = '10.164.0.31' geo_postgresql['sql_user_password'] = '' geo_secondary['db_fdw'] = true geo_postgresql['fdw_external_password'] = '' geo_postgresql['md5_auth_cidr_addresses'] = ['10.164.0.30/32'] geo_postgresql['trust_auth_cidr_addresses'] = %w(10.164.0.34/32 10.164.0.35/32) gitlab_rails['db_host'] = '10.164.0.30' gitlab_rails['auto_migrate'] = false alertmanager['enable'] = false consul['enable'] = false gitaly['enable'] = false gitlab_monitor['enable'] = false gitlab_workhorse['enable'] = false nginx['enable'] = false node_exporter['enable'] = false pgbouncer_exporter['enable'] = false postgresql['enable'] = false prometheus['enable'] = false redis['enable'] = false redis_exporter['enable'] = false repmgr['enable'] = false sidekiq['enable'] = false unicorn['enable'] = false geo_secondary['auto_migrate'] = false
Edited by Fabian Zimmer