project_repository_storage_moves API for archive projects fails

Summary

When project_repository_storage_moves API is used on archive projects the move process is executed, but it fails to change the Gitaly storage name for the repository.

The assumption is that this is due to the read only nature of the repository the storage is not updated.

Steps to reproduce

  1. Have a Gitaly cluster
  2. Create a project on local/nfs storage, and archive it.
  3. Execute API call to migrate it. It will fail
  4. Check praefect and gitaly repository paths, project is moved.
  5. Check Project in admin view, and see still on local/nfs storage
  6. Check logs, see move fails.

Example Project

What is the current bug behavior?

Project gets transfer but it is marked as failed and it does not update the repository storage.

What is the expected correct behavior?

Either project should not be transferred, or the project should be unarchive automatically before transfer, and then archived back.

Relevant logs and/or screenshots

praefect.log

api_json.log

gitaly1.json

gitaly2.json

gitaly3.json

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

```
root@app:~# sudo gitlab-rake gitlab:env:info

System information
System:         Ubuntu 20.04
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.7.5p203
Gem Version:    3.1.4
Bundler Version:2.1.4
Rake Version:   13.0.6
Redis Version:  6.2.6
Sidekiq Version:6.4.0
Go Version:     unknown

GitLab information
Version:        14.8.2-ee
Revision:       20a7fdf52c9
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     12.7
URL:            https://app.gitaly.gcp.gitlabsandbox.net
HTTP Clone URL: https://app.gitaly.gcp.gitlabsandbox.net/some-group/some-project.git
SSH Clone URL:  git@app.gitaly.gcp.gitlabsandbox.net:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: group_saml, saml_1, saml_2

GitLab Shell
Version:        13.23.2
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
- gitaly:       /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

```

Results of GitLab application Check

Expand for output related to the GitLab application check 


root@app:~#  sudo gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.23.2 ? ... OK (13.23.2)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK
gitaly ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
2/1 ... yes
1/2 ... yes
1/3 ... yes
1/4 ... yes
1/5 ... yes
1/6 ... yes
1/7 ... yes
1/8 ... yes
1/10 ... yes
21/11 ... yes
1/12 ... yes
18/14 ... yes
1/15 ... yes
1/16 ... yes
1/17 ... yes
1/18 ... yes
1/19 ... yes
1/20 ... yes
2/21 ... yes
1/22 ... yes
1/23 ... yes
1/24 ... yes
1/25 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.5)
Git user has default SSH configuration? ... yes
Active users: ... 3
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

groupgitaly ~"Gitaly Cluster" Gitaly Customer Issue customer

Edited by Gerardo Gutierrez