Skip to content
Snippets Groups Projects

Prevent Gitaly storages from using the same path

Merged Will Chandler (ex-GitLab) requested to merge wc/no-dup-gitaly-storages into master

What does this MR do?

Prevent Gitaly storages from using the same path

Gitaly is making a breaking change with v17.0 to prevent multiple storages from sharing the same local path. This is being done as part of the work to add a write-ahead log to Gitaly.

Validate that Gitaly's config does not have than one storage using the same path.

Related issues

gitaly#5598 (closed)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

  • MR title and description are up to date, accurate, and descriptive.
  • MR targeting the appropriate branch.
  • Latest Merge Result pipeline is green.
  • When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow.

For GitLab team members

If you don't have access to this, the reviewer should trigger these jobs for you during the review process.

  • The manual Trigger:ee-package jobs have a green pipeline running against latest commit.
  • If config/software or config/patches directories are changed, make sure the build-package-on-all-os job within the Trigger:ee-package downstream pipeline succeeded.
  • If you are changing anything SSL related, then the Trigger:package:fips manual job within the Trigger:ee-package downstream pipeline must succeed.
  • If CI configuration is changed, the branch must be pushed to dev.gitlab.org to confirm regular branch builds aren't broken.

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes.
  • Documentation created/updated.
  • Tests added.
  • Integration tests added to GitLab QA.
  • Equivalent MR/issue for the GitLab Chart opened.
  • Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.
Edited by Will Chandler (ex-GitLab)

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Gabriel Mazetto
  • added 1 commit

    • 4b500efe - Apply 1 suggestion(s) to 1 file(s)

    Compare with previous version

  • added 1 commit

    Compare with previous version

  • requested review from @brodock

  • Gabriel Mazetto approved this merge request

    approved this merge request

  • added 1 commit

    Compare with previous version

  • added 1 commit

    • e29a335b - Fix path and error assertion

    Compare with previous version

  • added 1 commit

    • 53a378fb - Don't modify original config object

    Compare with previous version

  • requested review from @rmarshall

    • Resolved by Robert Marshall

      One quibble over the emitted message and a concern about hardlinks.

      Super large projects (for example, mirrors) use hardlinks to retain paths without data on disk twice or symlink issues in some tools.

      I did not see in the realpath source where they account for hardlinks.

  • added 1 commit

    • 7c0e6cfc - Convert error to use 'path' instead of 'realpath'

    Compare with previous version

  • requested review from @rmarshall

  • added 1 commit

    Compare with previous version

    • Resolved by Robert Marshall

      Practical tests

      1. gitlab.rb
        external_url 'http://EXAMPLE.TLD'
        
        gitaly['configuration'] = {
          storage: [
            {
              name: 'default',
              path: '/var/opt/gitlab/git-data/my-favorite-repo-01',
            },
            {
              name: 'sneaksy',
              path: '/var/opt/gitlab/git-data/my-favorite-repo-01'
            }
          ]
        }
      2. First gitlab-ctl reconfigure at installation
        Running handlers:
        [2024-05-09T21:57:21+00:00] ERROR: Running exception handlers
        There was an error running gitlab-ctl reconfigure:
        
        One or more Gitaly storages are accessible by multiple filesystem paths:
          /var/opt/gitlab/git-data/my-favorite-repo-01: default, sneaksy
      3. Changed my gitlab.rb
        external_url 'http://EXAMPLE.TLD'
        
        gitaly['configuration'] = {
          storage: [
            {
              name: 'default',
              path: '/var/opt/gitlab/git-data/my-favorite-repo-01',
            },
            {
              name: 'sneaksy',
              path: '/var/opt/gitlab/git-data/my-favorite-repo-02'
            }
          ]
        }
      4. Ran gitlab-ctl reconfigure. It worked fine.
      5. Changed my gitlab.rb back to test after working gitlab-ctl reconfigure.
        external_url 'http://EXAMPLE.TLD'
        
        gitaly['configuration'] = {
          storage: [
            {
              name: 'default',
              path: '/var/opt/gitlab/git-data/my-favorite-repo-01',
            },
            {
              name: 'sneaksy',
              path: '/var/opt/gitlab/git-data/my-favorite-repo-01'
            }
          ]
        }
      6. This gave the same error message.
        There was an error running gitlab-ctl reconfigure:
        
        One or more Gitaly storages are accessible by multiple filesystem paths:
          /var/opt/gitlab/git-data/my-favorite-repo-01: default, sneaksy

      Functionality appears to work.

      Other things

      I noticed that when I set the storages, they were not automatically created even though they were on the local filesystem.

      Is that by design?

      ls -la /var/opt/gitlab/git-data
      total 12
      drwx------  3 git  git  4096 May  9 21:58 .
      drwxr-xr-x 21 root root 4096 May  9 22:02 ..
      drwxrws---  2 git  git  4096 May  9 21:58 repositories
  • Robert Marshall approved this merge request

    approved this merge request

  • Robert Marshall resolved all threads

    resolved all threads

  • mentioned in commit d0eba1b5

  • Balasankar 'Balu' C mentioned in merge request !7600 (merged)

    mentioned in merge request !7600 (merged)

  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Please register or sign in to reply
    Loading