Sign-off on enabling hashed storage for GitLab.com
In %10.0 we introduced "hashed storage". When enabled, rather than storing projects on disk in a location determined by project.full_path
, they are now stored in @hashed/aa/bb/full-hash
. The hash is generated by SHAing the project ID.
This has several advantages, but chiefly it means that renaming projects no longer requires a racy mv
command to be run on the project (and wiki) repository. This is particularly important for Geo setups, but large installations such as GitLab.com benefit in their own right.
I've been asked to evaluate the hashed storage system and attempt to determine whether %10.0 is a good time to enable it for GitLab.com and the Geo testbed.
There is also the question of migrating existing repositories to hashed storage. The migration path is still in development - see https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/14067 - but I should evaluate that as well, as far as I'm able to.
Testing progress:
-
Migrating non-hashed repos (simple case) https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/14067 -
Creating new projects when hashed storage is enabled -
Renaming a hashed project https://gitlab.com/gitlab-org/gitlab-ce/issues/38202 -
Audit EE to make sure everywhere that should use disk_path
does! -
Hashed storage migration: check cache invalidation after the migration has completed -
Fix reserved migration paths to work with hashed storage https://gitlab.com/gitlab-org/gitlab-ce/issues/40289 -
rake gitlab:sidekiq:drop_post_receive does not work with hashed storage https://gitlab.com/gitlab-org/gitlab-ce/issues/40290 -
rake gitlab:cleanup:repositories
will remove all hashed-storage repositories https://gitlab.com/gitlab-org/gitlab-ce/issues/40291 -
Cannot import a Wiki from Bitbucket when hashed storage is enabled https://gitlab.com/gitlab-org/gitlab-ce/issues/40292 -
Cannot list tags in a repository mirror when hashed storage in use https://gitlab.com/gitlab-org/gitlab-ce/issues/40300 -
Renaming a group or user may not be safe with hashed storage https://gitlab.com/gitlab-org/gitlab-ce/issues/40293