Configure rails db_extra_config_command attribute (!7356) · Merge requests · GitLab.org / omnibus-gitlab

Andrew Patterson requested to merge configure-rails-extra-db-config-command into master Jan 16, 2024

What does this MR do?

Configure gitlab_rails['db_extra_config_command'] in gitlab.rb which is used to specify a script to retrieve rails database password versus keeping the unencrypted password in the gitlab.rb file or in a disk file.

The contents of this attribute will be added to the rails database.yml file like so:

The gitlab.rb:

gitlab_rails['db_extra_config_command'] = '/opt/database-config.sh'

Resulting entry in database.yml

production:
                config_command: '/opt/database-config.sh'
                main:
.
.
.

There is no modification to how we generate default passwords for gitlab-rails['db_password'] as gitlab_rails['db_extra_config_command']` may be used for other than just credentials.

Related issues

Related #6274 (closed)
gitlab!141818 (merged)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

MR title and description are up to date, accurate, and descriptive.
MR targeting the appropriate branch.
Latest Merge Result pipeline is green.
When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow.

For GitLab team members

If you don't have access to this, the reviewer should trigger these jobs for you during the review process.

The manual Trigger:ee-package jobs have a green pipeline running against latest commit.
If config/software or config/patches directories are changed, make sure the build-package-on-all-os job within the Trigger:ee-package downstream pipeline succeeded.
If you are changing anything SSL related, then the Trigger:package:fips manual job within the Trigger:ee-package downstream pipeline must succeed.
If CI configuration is changed, the branch must be pushed to dev.gitlab.org to confirm regular branch builds aren't broken.

Expected (please provide an explanation if not completing)

Test plan indicating conditions for success has been posted and passes.
Documentation created/updated.
Tests added.
Integration tests added to GitLab QA.
Equivalent MR/issue for the GitLab Chart opened.
Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.

Test plan

Green pipeline including triggering Trigger::ee-package. Pipeline at https://gitlab.com/gitlab-org/omnibus-gitlab/-/pipelines/1142077455 fails QA decomposition-multiple-db jobs. Do our QA tests need to be modified to handle the new database.yml attribute?

Install and reconfigure using database defaults. Check for config_command: nil in database.yml file:

docker compose file

version: '3.8'

name: 'gitlab-test'

x-default_settings: &default_settings
  image: "${GITLAB_IMAGE:-registry.gitlab.com/gitlab-org/omnibus-gitlab/gitlab-ee:configure-rails-extra-db-config-command}"
  pull_policy: always
  privileged: true
  shm_size: '256m'
  restart: always

services:

  postgresql:
    <<: *default_settings
    image: postgres:14
    environment:
    - POSTGRES_USER=gitlab
    - POSTGRES_PASSWORD=secret_password
    - POSTGRES_DB=gitlabhq_production
    ports:
      - '5432:5432'
    networks:
      default:
        ipv4_address: 10.0.0.2
    healthcheck:
      test: pg_isready -U gitlab -d gitlabhq_production
      interval: 10s
      timeout: 3s
      retries: 3

  gitlab:
    <<: *default_settings
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url "http://10.0.0.3"

        gitlab_rails['db_database'] = "gitlabhq_production"
        gitlab_rails['db_username'] = "gitlab"
        gitlab_rails['db_password'] = "secret_password"
        gitlab_rails['db_host'] = '10.0.0.2'

        # Disable PG
        postgresql['enable'] = false
    ports:
      - "80:80"
      - "443:443"
      - "22:22"
    networks:
      default:
        ipv4_address: 10.0.0.3
    healthcheck:
      disable: true
    depends_on:
      postgresql:
        condition: service_healthy

networks:
  default:
    ipam:
      config:
        - subnet: 10.0.0.0/24

Gitlab container comes up with no issue.

$ docker exec -it  gitlab-test-gitlab-1 gitlab-ctl status             
run: alertmanager: (pid 1529) 408s; run: log: (pid 1363) 423s
run: gitaly: (pid 1479) 410s; run: log: (pid 575) 500s
run: gitlab-exporter: (pid 1489) 409s; run: log: (pid 815) 439s
run: gitlab-kas: (pid 597) 498s; run: log: (pid 620) 495s
run: gitlab-workhorse: (pid 1412) 410s; run: log: (pid 778) 448s
run: logrotate: (pid 498) 516s; run: log: (pid 506) 515s
run: nginx: (pid 1448) 410s; run: log: (pid 791) 446s
run: prometheus: (pid 1498) 409s; run: log: (pid 1342) 427s
run: puma: (pid 650) 461s; run: log: (pid 657) 460s
run: redis: (pid 514) 510s; run: log: (pid 523) 509s
run: redis-exporter: (pid 1491) 409s; run: log: (pid 1104) 433s
run: sidekiq: (pid 669) 455s; run: log: (pid 689) 454s
run: sshd: (pid 37) 526s; run: log: (pid 36) 526s

Correct config_command entry in database.yml file.

$ docker exec -it  gitlab-test-gitlab-1 head /opt/gitlab/embedded/service/gitlab-rails/config/database.yml
# This file is managed by gitlab-ctl. Manual changes will be
# erased! To change the contents below, edit /etc/gitlab/gitlab.rb
# and run `sudo gitlab-ctl reconfigure`.

production:
  config_command: nil
  main:
    adapter: postgresql
    encoding: unicode
    collation:

Set gitlab_rails['db_extra_config_command]

Bring up new docker compose using the above docker compose file.
Create configure command at /opt/database-config.sh in the gitlab-test-gitlab-1 container using the following content:
```
#!/bin/bash

cat << EOF
---
main:
  password: 'secret_password'
EOF
```
Set permissions on the configuration command:
```
chmod 755 /opt/database-config.sh
```

Edit the /etc/gitlab/gitlab.rb file and set:

gitlab_rails['db_password'] = nil
gitlab_rails['db_extra_config_command'] = '/opt/database-config.sh'

Run gitlab-ctl reconfigure and check status

$ gitlab-ctl reconfigure
$ gitlab-ctl status
ok: run: alertmanager: (pid 3602) 1s
ok: run: gitaly: (pid 3617) 0s
ok: run: gitlab-exporter: (pid 3653) 0s
ok: run: gitlab-kas: (pid 3663) 0s
ok: run: gitlab-workhorse: (pid 3678) 1s
ok: run: logrotate: (pid 3697) 0s
ok: run: nginx: (pid 3703) 1s
ok: run: prometheus: (pid 3735) 0s
ok: run: puma: (pid 3763) 1s
ok: run: redis: (pid 3768) 0s
ok: run: redis-exporter: (pid 3777) 1s
ok: run: sidekiq: (pid 3795) 0s
ok: run: sshd: (pid 3803) 1s
curl http://10.0.0.3
<html><body>You are being <a href="http://10.0.0.3/users/sign_in">redirected</a>.</body></html>

Check contents of database.yml to check if config_command is populated with the script path.

cat /opt/gitlab/embedded/service/gitlab-rails/config/database.yml | head -n 8
# This file is managed by gitlab-ctl. Manual changes will be
# erased! To change the contents below, edit /etc/gitlab/gitlab.rb
# and run `sudo gitlab-ctl reconfigure`.

production:
  config_command: "/opt/database-config.sh"
  main:
    adapter: postgresql

Remove old passwords from database.yml to make sure it is actually using the command.

$ sed -ri -e 's/.*password\:.*/    password:/' /opt/gitlab/embedded/service/gitlab-rails/config/database.yml

Restart gitlab and check operation.

$ gitlab-ctl restart
$ ok: run: alertmanager: (pid 4542) 0s
ok: run: gitaly: (pid 4559) 0s
ok: run: gitlab-exporter: (pid 4595) 0s
ok: run: gitlab-kas: (pid 4604) 1s
ok: run: gitlab-workhorse: (pid 4619) 0s
ok: run: logrotate: (pid 4639) 0s
ok: run: nginx: (pid 4645) 1s
ok: run: prometheus: (pid 4677) 0s
ok: run: puma: (pid 4705) 0s
ok: run: redis: (pid 4710) 1s
ok: run: redis-exporter: (pid 4719) 0s
ok: run: sidekiq: (pid 4733) 0s
ok: run: sshd: (pid 4744) 0s
$ curl http://10.0.0.3
<html><body>You are being <a href="http://10.0.0.3/users/sign_in">redirected</a>.</body></html>

Edited Jan 18, 2024 by Andrew Patterson

Configure rails db_extra_config_command attribute