Skip to content

Redis node fails to upgrade from 15.8.1 to 15.8.3

Summary

A customer ran into an issue when attempting to upgrade their Omnibus Redis node from 15.8.1 to 15.8.3. The Redis node doesn’t have a /var/opt/gitlab/gitlab-rails/etc directory so they get a EnclosingDirectoryDoesNotExist error.

Relevant logs

  • yum update failure:
Downloading Packages:
gitlab-ee-15.8.3-ee.0.el8.x86\_64.rpm                                                74 MB/s | 1.2 GB     00:16
-------------------------------------------------------------------------------------------------------------------
Total                                                                               74 MB/s | 1.2 GB     00:16
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                           1/1
  Running scriptlet: gitlab-ee-15.8.3-ee.0.el8.x86\_64                                                          1/2
Malformed configuration JSON file found at /opt/gitlab/embedded/nodes/hq2-git-redis00.paycomhq.com.json.
This usually happens when your last run of \`gitlab-ctl reconfigure\` didn't complete successfully.
This file is used to check if any of the unsupported configurations are enabled,
and hence require a working reconfigure before upgrading.
Please run \`sudo gitlab-ctl reconfigure\` to fix it and try again.
error: %prein(gitlab-ee-15.8.3-ee.0.el8.x86\_64) scriptlet failed, exit status 1Error in PREIN scriptlet in rpm package gitlab-ee
  Verifying        : gitlab-ee-15.8.3-ee.0.el8.x86\_64                                                          1/2
  Verifying        : gitlab-ee-15.8.1-ee.0.el8.x86\_64                                                          2/2
Installed products updated.Failed:
  gitlab-ee-15.8.1-ee.0.el8.x86\_64                                          gitlab-ee-15.8.3-ee.0.el8.x86\_64Error: Transaction failed
  • Ran gitlab-ctl reconfigure as suggested in the output gave this error:
Running handlers complete
\[2023-02-25T09:52:49-06:00\] ERROR: Exception handlers complete
Infra Phase failed. 4 resources updated in 01 minutes 18 seconds
\[2023-02-25T09:52:49-06:00\] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out
\[2023-02-25T09:52:49-06:00\] FATAL: ---------------------------------------------------------------------------------------
\[2023-02-25T09:52:49-06:00\] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
\[2023-02-25T09:52:49-06:00\] FATAL: ---------------------------------------------------------------------------------------
\[2023-02-25T09:52:49-06:00\] FATAL: Chef::Exceptions::EnclosingDirectoryDoesNotExist: templatesymlink\[Create a gitlab\_suggested\_reviewers\_secret and create a symlink to Rails root\] (gitlab-ee::suggested\_reviewers line 30) had an error: Chef::Exceptions::EnclosingDirectoryDoesNotExist: template\[/var/opt/gitlab/gitlab-rails/etc/gitlab\_suggested\_reviewers\_secret\] (gitlab-ee::suggested\_reviewers line 36) had an error: Chef::Exceptions::EnclosingDirectoryDoesNotExist: Parent directory /var/opt/gitlab/gitlab-rails/etc does not exist.
\[sai.ram.nukala@hq2-git-redis00 ~\]$ sudo cd /var/opt/gitlab/gitlab-rails/etc
/bin/cd: line 2: cd: /var/opt/gitlab/gitlab-rails/etc: No such file or directory
\[sai.ram.nukala@hq2-git-redis00 ~\]$ sudo cd /var/opt/gitlab/gitlab-rails
/bin/cd: line 2: cd: /var/opt/gitlab/gitlab-rails: No such file or directory

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb` 

external_url ''
# Listen on all addresses
redis['bind'] = '0.0.0.0'
# Set the defaul port, must be set.
redis['port'] = 
# Set password, as in the secret `gitlab-redis` populated in Kubernetes
redis['password'] = 'REDIS_PASSWORD' # On Deploy Replace

## Configure Geo
gitlab_rails['geo_node_name'] = '<GEO_NODE_NAME>'


# Monitoring
node_exporter['enable'] = true
node_exporter['listen_address'] = '0.0.0.0:'
redis_exporter['enable'] = true
redis_exporter['listen_address'] = '0.0.0.0:'


## Disable everything else
gitlab_rails['enable'] = false
sidekiq['enable'] = false
puma['enable']=false
registry['enable'] = false
gitaly['enable'] = false
gitlab_workhorse['enable'] = false
nginx['enable'] = false
prometheus_monitoring['enable'] = false
postgresql['enable'] = false
gitlab_kas['enable'] = false

Possible Workarounds

Create the folder /var/opt/gitlab/gitlab-rails.

Edited by Elif Munn