Custom Gitaly storage directory/ies not created from 16.0

Summary

Customers have reported that builds of Gitaly servers are no long creating the repositories data directory since GitLab 16.0, as a result of the change in the way Gitaly configuration is structured in gitlab.rb

Gitaly logs:

invalid config: storage path \"/gitlab-data/git-data/repositories\" for storage \"default\" doesn't exist

Workaround

In this example, the location has been changed from /var/opt/gitlab/git-data/repositories. It's common for a separate filesystem to be mounted elsewhere, such as on /gitlab-data in this example:

gitaly['configuration'] = {
  storage: [
    {
      name: 'default',
      path: '/gitlab-data/git-data/repositories',
    },
  ],
}

In GitLab 15.11 and earlier, this could be specified as:

git_data_dirs({
  'default' => {
    'path' => '/gitlab-data/git-data/repositories'
  },
})

Existing servers, built with 15.11 or earlier using the previous configuration structure, aren't affected.

On new servers:

1. Create the path, once the filesystem is mounted (if required):

   mkdir -p /gitlab-data/git-data/repositories

2. Set the permissions correctly:

       # drwx------  git git
   chmod 0700 /gitlab-data/git-data
   chown git:git /gitlab-data/git-data
       # drwxrws---  git git
   chmod 2770 /gitlab-data/git-data/repositories
   chown git:git /gitlab-data/git-data/repositories

Steps to reproduce

Additional storages or modify the existing storage to use a non-default location.

  storage: [
    {
      # gitaly['storage'][<index>]['name']
      #
      # git_data_dirs[<name>]. The storage name was configured as a key in the map.
      name: ...,
      # gitaly['storage'][<index>]['path']
      #
      # git_data_dirs[<name>]['path']. Use the value from git_data_dirs[<name>]['path'] and append '/repositories' to it.
      #
      # For example, if the path in 'git_data_dirs' was '/var/opt/gitlab/git-data', use
      # '/var/opt/gitlab/git-data/repositories'. The '/repositories' extension was automatically
      # appended to the path configured in `git_data_dirs`.
      path: ...,
    },
  ],

Run gitlab-ctl reconfigure

What is the current bug behavior?

gitlab-ctl reconfigure doesn't create the custom directory structure, set the right permissions and ownership.

What is the expected correct behavior?

gitlab-ctl reconfigure creates the custom directory structure, and set the right permissions and ownership.

Relevant logs

Relevant logs

Details of package version

16.0.0 and later