Pre-existing initial_root_password in /etc/gitlab/gitlab.rb causes cloud-init to fail

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Summary

We are running Gitlab using the official AWS AMI on EC2. Using a value for initial_root_password in /etc/gitlab/gitlab.rb causes the Gitlab-provided per-instance/gitlab script for cloud-init to fail during gitlab-ctl reconfigure, leaving a newly provisioned instance unconfigured and requiring manual intervention.

Steps to reproduce

• Ensure that /etc/gitlab/gitlab.rb is present and contains a setting for initial_root_password.
• Manually run /var/lib/cloud/scripts/per-instance/gitlab. In my testing, the behavior was reproducible at any point if the setting was present in gitlab.rb.

Example Project

n/a

What is the current bug behavior?

I order to avoid embedding secrets in userdata, we template /etc/gitlab/gitlab.rb with values from SSM Parameter Store. Among these values is initial_root_password. This works correctly, and the setting is pre-filled with an alnum-only 32 character string. After starting up, a manual gitlab-ctl reconfigure works correctly, as does the password login.

However, the script /var/lib/cloud/scripts/per-instance/gitlab contains the following:

EXISTING_ROOT_PASSWORD=$(sudo grep "^gitlab_rails.*initial_root_password.*" /etc/gitlab/gitlab.rb | cut -d '=' -f2- | xargs)
if [ -z "${EXISTING_ROOT_PASSWORD}" ] && [ -z "${GITLAB_ROOT_PASSWORD}" ]; then
  GITLAB_ROOT_PASSWORD=$(curl http://169.254.169.254/latest/meta-data/instance-id)
fi

sudo GITLAB_ROOT_PASSWORD=${GITLAB_ROOT_PASSWORD} gitlab-ctl reconfigure

This contains a logic error: If EXISTING_ROOT_PASSWORD is set, GITLAB_ROOT_PASSWORD is set to an empty string. In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab_rails.rb, the root password is validated as such:

      # Environment variable gets priority over gitlab.rb setting
      Gitlab['gitlab_rails']['initial_root_password'] = ENV['GITLAB_ROOT_PASSWORD'] || Gitlab['gitlab_rails']['initial_root_password']
[...]
      raise 'initial_root_password: Length is too short, minimum is 8 characters' if Gitlab['gitlab_rails']['initial_root_password'] && Gitlab['gitlab_rails']['initial_root_password'].length < 8

Since GITLAB_ROOT_PASSWORD is an empty string and not nil or false, it takes precedence over the setting in gitlab.rb. This results in the following error:

Running handlers:
[2024-01-05T11:27:53+00:00] ERROR: Running exception handlers
There was an error running gitlab-ctl reconfigure:

initial_root_password: Length is too short, minimum is 8 characters

What is the expected correct behavior?

If provided in gitlab.rb, the user-defined password should be used.

Relevant logs and/or screenshots

No further logs needed, I think.

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:         Ubuntu 20.04
Current User:   git
Using RVM:      no
Ruby Version:   3.1.4p223
Gem Version:    3.4.22
Bundler Version:2.4.22
Rake Version:   13.0.6
Redis Version:  7.0.14
Sidekiq Version:6.5.12
Go Version:     unknown

GitLab information
Version:        16.7.0
Revision:       0fb7483d0df
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     13.12
URL:            https://git.redacted.com
HTTP Clone URL: https://git.redacted.com/some-group/some-project.git
SSH Clone URL:  git@git.redacted.com:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        14.32.0
Repository storages:
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Gitaly
- default Address:      unix:/var/opt/gitlab/gitaly/gitaly.socket
- default Version:      16.7.0
- default Git Version:  2.42.0

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.32.0 ? ... OK (14.32.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
Tables are truncated? ... skipped
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Cable config exists? ... yes
Resque config exists? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
2/1 ... yes
1/2 ... yes
8/3 ... yes
11/4 ... yes
11/6 ... yes
13/7 ... yes
13/8 ... yes
13/9 ... yes
13/10 ... yes
13/11 ... yes
13/12 ... yes
13/13 ... yes
13/14 ... yes
13/15 ... yes
20/16 ... yes
13/17 ... yes
14/18 ... yes
33/19 ... yes
13/20 ... yes
14/21 ... yes
23/22 ... yes
21/23 ... yes
21/24 ... yes
14/25 ... yes
21/26 ... yes
18/27 ... yes
8/28 ... yes
13/32 ... yes
33/33 ... yes
8/34 ... yes
14/35 ... yes
33/36 ... yes
33/37 ... yes
33/38 ... yes
14/40 ... yes
33/41 ... yes
33/42 ... yes
18/43 ... yes
20/44 ... yes
31/45 ... yes
14/46 ... yes
33/47 ... yes
33/48 ... yes
14/49 ... yes
14/50 ... yes
31/51 ... yes
89/52 ... yes
33/53 ... yes
89/54 ... yes
18/55 ... yes
31/56 ... yes
33/57 ... yes
33/58 ... yes
33/59 ... yes
104/61 ... yes
104/62 ... yes
104/63 ... yes
116/66 ... yes
116/70 ... yes
14/71 ... yes
13/72 ... yes
104/73 ... yes
104/74 ... yes
113/75 ... yes
14/76 ... yes
104/77 ... yes
33/78 ... yes
104/79 ... yes
104/80 ... yes
104/81 ... yes
104/82 ... yes
104/83 ... yes
142/84 ... yes
142/85 ... yes
148/86 ... yes
148/87 ... yes
116/88 ... yes
14/89 ... yes
159/90 ... yes
104/91 ... yes
116/92 ... yes
104/93 ... yes
104/94 ... yes
104/95 ... yes
33/96 ... yes
104/97 ... yes
142/98 ... yes
148/99 ... yes
13/100 ... yes
139/101 ... yes
142/104 ... yes
8/105 ... yes
8/106 ... yes
14/107 ... yes
33/108 ... yes
14/109 ... yes
20/110 ... yes
20/111 ... yes
8/113 ... yes
8/114 ... yes
8/115 ... yes
104/116 ... yes
159/120 ... yes
23/121 ... yes
142/124 ... yes
14/125 ... yes
180/128 ... yes
14/129 ... yes
231/130 ... yes
23/131 ... yes
14/132 ... yes
14/133 ... yes
8/134 ... yes
104/135 ... yes
180/136 ... yes
243/137 ... yes
8/138 ... yes
245/139 ... yes
18/140 ... yes
18/141 ... yes
8/142 ... yes
33/143 ... yes
142/144 ... yes
8/145 ... yes
14/146 ... yes
14/147 ... yes
266/148 ... yes
142/149 ... yes
13/150 ... yes
20/151 ... yes
273/153 ... yes
Redis version >= 6.0.0? ... yes
Ruby version >= 3.0.6 ? ... yes (3.1.4)
Git user has default SSH configuration? ... yes
Active users: ... 65
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes

https://github.com/gitlabhq/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/libraries/gitlab_rails.rb#L66

Either change the evaluation here to disallow empty strings for the env or set the env to EXISTING_ROOT_PASSWORD like so:

EXISTING_ROOT_PASSWORD=$(sudo grep "^gitlab_rails.*initial_root_password.*" /etc/gitlab/gitlab.rb | cut -d '=' -f2- | xargs)
if [ -z "${EXISTING_ROOT_PASSWORD}" ] && [ -z "${GITLAB_ROOT_PASSWORD}" ]; then
  GITLAB_ROOT_PASSWORD=$(curl http://169.254.169.254/latest/meta-data/instance-id)
else
  GITLAB_ROOT_PASSWORD="${GITLAB_ROOT_PASSWORD:-${EXISTING_ROOT_PASSWORD}}"
fi

(This is a sample fix - depending on what your actual intended outcome priorities are, this may not do what you want.)