Registry migration fails to restart service when using sudo -u registry

What

The error we get os fatal: unable to open current directory: access denied when running gitlab-ctl registry-database migrate up with the registry user from a directory that the registry does not have access to. This happens because after migrations are run, the registry attempts to restart the runit service by forking a process which inherits a current directory that it does not have access to.

When we run on a VM with Ubuntu or SELinux, for instance, we're usually logged in with a custom user profile. And we're usually located on this users $HOME directory, which is usually a directory that GitLab has not rights on. So Runit ends up failing to open the current directory.

This is usually not a problem with our Docker deployments, since we don't have sudo and we use su - registry instead, which moves the command to be run from the users home directory:

root@f2d788c1c19e:/home/ubuntu# su - registry bash -c 'pwd'
/var/opt/gitlab/registry

So the problem only happens with sudo -u registry.

More context

Here's what was happening. When we run:

sudo -u registry bash -c 'gitlab-ctl registry-database migrate up --skip-post-deployment'

If the command was in /home/jcunha (my $HOME directory on my Ubuntu VM), after the migration is completed, the command attempts to restart the registry service. The problem occurs inside run_sv_command_for_service, which is defined in the omnibus-ctl lib:

def run_sv_command_for_service(sv_cmd, service_name)
  if service_enabled?(service_name)
    status = run_command("#{base_path}/init/#{service_name} #{sv_cmd}")
    # ...
  end
end

This calls run_command, which uses system():

def run_command(command)
  system(command)
  $?
end

When system() spawns the /opt/gitlab/init/registry start script, it:

  1. Forks a new process
  2. The child process inherits the current working directory (/home/jcunha)
  3. The sv/runit scripts try to access or verify the current directory
  4. But the registry user doesn't have permission to access /home/jcunha ❌

This causes the "unable to open current directory: access denied" error.

Example Logs

# Fatal error because after the migration happens the registry tries to restart the service from my users $HOME dir:
Alexand@jcunha-test-omnibus:~$ sudo -u registry bash -c 'gitlab-ctl registry-database migrate up --skip-post-deployment'
Running migrate up
Executing command:
/opt/gitlab/embedded/bin/registry database migrate up -s /var/opt/gitlab/registry/config.yml
OK: applied 0 pre-deployment migration(s), 0 post-deployment migration(s) and 0 background migration(s) in 0.010s
Starting service registry
fatal: unable to open current directory: access denied

# Confirmation that the problem is about the restart, not about the migration
Alexand@jcunha-test-omnibus:~$ sudo -u registry bash -c 'gitlab-ctl restart registry'
fatal: unable to open current directory: access denied

# Confirmation that if we cd into another directory, the problem does not happen
Alexand@jcunha-test-omnibus:~$ sudo -u registry bash -c 'cd /var/opt/gitlab && gitlab-ctl registry-database migrate up --skip-post-deployment'
Running migrate up
Executing command:
/opt/gitlab/embedded/bin/registry database migrate up -s /var/opt/gitlab/registry/config.yml
OK: applied 0 pre-deployment migration(s), 0 post-deployment migration(s) and 0 background migration(s) in 0.010s

Proposal

Force the gitlab-ctl registry-database commands to do a temporary cd into a directory that runit has access to.

Edited by João Alexandre Cunha