In Cloud Native GitLab the Rake task gitlab:uploads:migrate_to_local:all loses uploads

Summary

Similar to #231297 (closed)

Following https://docs.gitlab.com/13.12/ee/administration/raketasks/uploads/migrate.html#migrate-to-local-storage ran gitlab-rake "gitlab:uploads:migrate_to_local:all" which led to projects and users avatars be missing.

Steps to reproduce

  1. Run gitlab-rake "gitlab:uploads:migrate_to_local:all", in a while check uploads table using select query from KB page, which showed that objectstg and filesystem store stats reversed = seems task finished ok.
  2. Check UI, observe missing avatars
  3. Run gitlab-rake "gitlab:uploads:migrate:all", nothing changes in uploads table store stats.
  4. Browse to minio objectstg UI page, e.g. minio/gitlab-uploads/user/avatar/, observe missing now folders.

What is the current bug behavior?

Uploads are lost.

What is the expected correct behavior?

No loss of images, Rake tasks detects (maybe) faulty configuration and errors out detailing the mis-configuration.

Reverse task 'migrate' is able to restore/migrate uploads back to objectstg.

Omnibus instance where we're migrating to has the lost uploads files from a backup, but they are not shown in UI as db has incorrect paths for files. E.g. a db path refers to "uploads/-/system/user/avatar/3620/avatar.png", but when hitting https://FQDN/uploads/-/system/user/avatar/3620/avatar.png it redirects to https://FQDNuser/avatar/3620/avatar.png which obviously is not resolved. Any help to fix the uploads, e.g. come up with a sql procedure, which 'd point to correct filesystem paths for all the entries is appreciated.

Also in addition to old app ver, we have complexity: omnibus installation being migrated - we use Azure object storage, but it seems not to be really supported yet in 13.2. https://docs.gitlab.com/ee/administration/object_storage.html#azure-blob-storage

So we are thinking of setting up minio / using AWS storage, restore uploads files and db table from the backup, then run migrate_to_local rake task, then disable object storage for uploads.

Relevant logs and/or screenshots

stats before running task

gitlabhq_production=# SELECT count(*) AS total, sum(case when store = '1' then 1 else 0 end) AS filesystem, sum(case when store = '2' then 1 else 0 end) AS objectstg FROM uploads;
 total | filesystem | objectstg
-------+------------+-----------
 11179 |          8 |     11171
(1 row)
  uploads:
    enabled: true
    object_store:
      enabled: true
      remote_directory: gitlab-uploads
      direct_upload: true
      background_upload: false
      proxy_download: true
      connection:
        provider: AWS
        region: us-east-1
        aws_access_key_id: \"<%= File.read('/etc/gitlab/minio/accesskey').strip.dump[1..-2] %>\"
        aws_secret_access_key: \"<%= File.read('/etc/gitlab/minio/secretkey').strip.dump[1..-2] %>\"
        host: minio.labs.nuance.com
        endpoint: http://gitlab-minio-svc:9000
        path_style: true

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
``
git@gitlab-task-runner-6c56bbb5d9-2jh8j:/$ gitlab-rake gitlab:env:info

System information
System:
Current User:	git
Using RVM:	no
Ruby Version:	2.6.6p146
Gem Version:	2.7.10
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	unknown
Git Version:	unknown
Sidekiq Version:5.2.9
Go Version:	unknown

GitLab information
Version:	13.2.0
Revision:
Directory:	/srv/gitlab
DB Adapter:	PostgreSQL
DB Version:	11.7
URL:		https://FQDN
HTTP Clone URL:	https://FQDN/some-group/some-project.git
SSH Clone URL:	git@FQDN:some-group/some-project.git
Using LDAP:	yes
Using Omniauth:	no

GitLab Shell
Version:	13.3.0
Repository storage paths:
- default: 	/var/opt/gitlab/repo
GitLab Shell path:		/home/git/gitlab-shell
Git:		/usr/bin/git
``

Results of GitLab application Check

Expand for output related to the GitLab application check 
helm installation, but below output from omnibus instance where we're migrating to:
check.out

Possible fixes

Edited by 🤖 GitLab Bot 🤖