rollback_to_legacy causes repos to 404
Summary
Can't migrate repos back to legacy storage
Steps to reproduce
sudo gitlab-rake gitlab:storage:migrate_to_hashed
# wait for the migration and test a few repos
sudo gitlab-rake gitlab:storage:rollback_to_legacy
# all repos throw 404 now, as they haven't been really migrated. Only the DB entry was changedWhat is the current bug behavior?
Migrating from hashed_storage back to legacy storage causes all repositories to 404. New repositories created with legacy storage work fine.
What is the expected correct behavior?
Repositories should be migrated back to legacy storage just fine
Relevant logs and/or screenshots
logs look good
{
"severity": "INFO",
"time": "2020-03-16T14:55:44.495Z",
"class": "HashedStorage::ProjectRollbackWorker",
"retry": 3,
"queue": "hashed_storage:hashed_storage_project_rollback",
"queue_namespace": "hashed_storage",
"jid": "d2fc6633688f265a87f4ab76",
"created_at": "2020-03-16T14:55:43.283Z",
"meta.caller_id": "HashedStorage::RollbackerWorker",
"correlation_id": "9a2f43d4bd96097bbe69692d88ac1209",
"enqueued_at": "2020-03-16T14:55:43.283Z",
"pid": 24256,
"message": "HashedStorage::ProjectRollbackWorker JID-d2fc6633688f265a87f4ab76: done: 1.209761 sec",
"job_status": "done",
"scheduling_latency_s": 0.001768,
"gitaly_calls": 4,
"gitaly_duration": 143.91,
"duration": 1.209761,
"cpu_s": 0.045272,
"completed_at": "2020-03-16T14:55:44.495Z",
"db_duration": 786.1539437435567,
"db_duration_s": 0.7861539437435567
}
Output of checks
all good
Results of GitLab environment info
Expand for output related to GitLab environment info
[z003s32w@frontend-dev-primary ~]$ sudo gitlab-rake gitlab:env:infoSystem information System: Current User: git Using RVM: no Ruby Version: 2.6.5p114 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.7 Git Version: 2.24.1 Sidekiq Version:5.2.7 Go Version: unknown
GitLab information Version: 12.8.2 Revision: 785e16f105a Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.7 URL: https://- HTTP Clone URL: https://-/some-group/some-project.git SSH Clone URL: git@-:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: saml
GitLab Shell Version: 11.0.0 Repository storage paths:
- default: /data/gitlab/git/repositories
- gitaly-02: /data/gitlab/git/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Possible fixes
I'm guessing that somehow this fails, but the repo is changed anyway: