Task gitlab:uploads:migrate_to_local:all loses uploads
Summary
Similar to #231297 (closed)
Following https://docs.gitlab.com/13.12/ee/administration/raketasks/uploads/migrate.html#migrate-to-local-storage ran
gitlab-rake "gitlab:uploads:migrate_to_local:all"
which led to projects and users avatars be missing.
Steps to reproduce
- Run
gitlab-rake "gitlab:uploads:migrate_to_local:all"
, in a while check uploads table using select query from KB page, which showed that objectstg and filesystem store stats reversed = seems task finished ok. - Check UI, observe missing avatars
- Run
gitlab-rake "gitlab:uploads:migrate:all"
, nothing changes in uploads table store stats. - Browse to minio objectstg UI page, e.g. minio/gitlab-uploads/user/avatar/, observe missing now folders.
What is the current bug behavior?
Uploads are lost.
What is the expected correct behavior?
No loss of images, Rake tasks detects (maybe) faulty configuration and errors out detailing the mis-configuration.
Reverse task 'migrate' is able to restore/migrate uploads back to objectstg.
Omnibus instance where we're migrating to has the lost uploads files from a backup, but they are not shown in UI as db has incorrect paths for files. E.g. a db path refers to "uploads/-/system/user/avatar/3620/avatar.png", but when hitting https://FQDN/uploads/-/system/user/avatar/3620/avatar.png it redirects to https://FQDNuser/avatar/3620/avatar.png which obviously is not resolved. Any help to fix the uploads, e.g. come up with a sql procedure, which 'd point to correct filesystem paths for all the entries is appreciated.
Also in addition to old app ver, we have complexity: omnibus installation being migrated - we use Azure object storage, but it seems not to be really supported yet in 13.2. https://docs.gitlab.com/ee/administration/object_storage.html#azure-blob-storage
So we are thinking of setting up minio / using AWS storage, restore uploads files and db table from the backup, then run migrate_to_local rake task, then disable object storage for uploads.
Relevant logs and/or screenshots
stats before running task
gitlabhq_production=# SELECT count(*) AS total, sum(case when store = '1' then 1 else 0 end) AS filesystem, sum(case when store = '2' then 1 else 0 end) AS objectstg FROM uploads;
total | filesystem | objectstg
-------+------------+-----------
11179 | 8 | 11171
(1 row)
uploads:
enabled: true
object_store:
enabled: true
remote_directory: gitlab-uploads
direct_upload: true
background_upload: false
proxy_download: true
connection:
provider: AWS
region: us-east-1
aws_access_key_id: \"<%= File.read('/etc/gitlab/minio/accesskey').strip.dump[1..-2] %>\"
aws_secret_access_key: \"<%= File.read('/etc/gitlab/minio/secretkey').strip.dump[1..-2] %>\"
host: minio.labs.nuance.com
endpoint: http://gitlab-minio-svc:9000
path_style: true
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
`` git@gitlab-task-runner-6c56bbb5d9-2jh8j:/$ gitlab-rake gitlab:env:info System information System: Current User: git Using RVM: no Ruby Version: 2.6.6p146 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: unknown Git Version: unknown Sidekiq Version:5.2.9 Go Version: unknown GitLab information Version: 13.2.0 Revision: Directory: /srv/gitlab DB Adapter: PostgreSQL DB Version: 11.7 URL: https://FQDN HTTP Clone URL: https://FQDN/some-group/some-project.git SSH Clone URL: git@FQDN:some-group/some-project.git Using LDAP: yes Using Omniauth: no GitLab Shell Version: 13.3.0 Repository storage paths: - default: /var/opt/gitlab/repo GitLab Shell path: /home/git/gitlab-shell Git: /usr/bin/git ``
Results of GitLab application Check
Expand for output related to the GitLab application check
helm installation, but below output from omnibus instance where we're migrating to: check.out