Hashed storage support for GitLab pages

When hashed storage is enabled, when a project is renamed it doesn't need to be moved on disk. We should update GitLab Pages with support for hashed storage too.

Proposal

Store pages for a project in a hashed location so that it doesn't need to be moved on disk when the project is renamed or transferred.

Links

https://gitlab.com/gitlab-org/gitlab-ce/issues/34489#note_69261016

@nick.thomas @DouweM created following discussion https://gitlab.com/gitlab-org/gitlab-ce/issues/34489#note_69261016 as an alternative to

We could consider disabling project renames when Pages is enabled

/cc @bikebilly @ayufan

The difficulty here is that the URL contains the project path, but to get to the path on disk we need to convert that path to the hashed form of the project ID. This mapping is contained in the SQL database, which the pages daemon has no access to.

If we start writing the pages artifacts to disk under the hierarchy determined by hashed storage, we need some way to access that mapping. We also need it to continue working when the SQL database isn't running.

We write config.json files into the -pages-path per-project. We could start writing the project's full_path into there, so -pages-root starts to look like:

shared/
  pages/
    xx/
      yy/
        abcdefg..../
          config.json # { "FullPath": "foo/bar" }

On project rename, we now just have to re-write the config.json file to update the mapping.

On startup, and at every configuration change, we process these files and build an in-memory mapping of the full_path to the directory. Obviously, it doesn't scale indefinitely - we already have problems relating to pages configuration - but that can be addressed separately. I don't think it makes the problem much worse.

The other difficulty, of course, is migrating the existing dataset, including rollback in case of bugs.

@nick.thomas Would it be an option to have use symlinks to map from full paths to hashed storage? On rename, we'd "just" need to update the symlink

@DouweM we could do that, but since the -pages-root is generally all one filesystem, I don't think this gets us any advantages over the straight mv we do at present.

The big advantage of putting the mapping in the config.json file is that we completely regenerate it every time pages is deployed. So in the event that the rename happens and the update is somehow missed, there is a complete self-service fix - just trigger another pages pipeline.

In both the mv and symlink cases, this self-service fix also "works", but leaves bad data behind on the filesystem.

changed milestone to %Awaiting further demand

This item has been automatically determined to have 4 or fewer upvotes and is, for now, being marked as awaiting further demand. If you feel this is incorrect, please comment on the issue and I'll be happy to take a look.

added [deprecated] Accepting merge requests label

added 1 deleted label and removed 1 deleted label

changed milestone to %Backlog

added 1 deleted label

changed milestone to %Backlog

added to epic &2320

removed 1 deleted label

marked this issue as related to gitlab-pages#355 (closed)

marked this issue as related to gitlab-foss#34489 (closed)

mentioned in epic &2320

marked this issue as related to #208135 (closed)

mentioned in issue #208135 (closed)

mentioned in issue #18754 (closed)

mentioned in issue gitlab-org/quality/pipeline-triage#31 (closed)

added PagesObject Storage label

I think we can close this in favor of &3901 (closed).

If anyone disagrees, please reopen!

closed