Skip to content

Rake task: Do not generate too long file names for uploads

Summary

After merging !88872 (merged) we should provide a rake task that centralizes and automates the given steps to unblock users who try to make a backup procedure. This MR provides this automation for backup and restore.

Steps to reproduce

One or more uploaded files with filenames longer than 255 characters. UNIX-based machines allow the creation of filenames with 255 bytes maximum, so this issue seems to be replicable only in machines using Windows as OS.

What is the current bug behavior?

The file name generated for a file uploaded via Gitlab UI is too long, and the backup script can't proceed with the backup as expected.

What is the expected correct behavior?

Backup and restore procedures should move forward when this error happens. In addition, the rake task should provide fixes for all the upload records, references, and filesystem.

Relevant logs and/or screenshots

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  and have a look at the known issues list:
    https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions-(FAQ)
  If the error persists, please report the
  following lines (removing any private
  info as necessary) to:
   s3tools-bugs@lists.sourceforge.net


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Invoked as: /usr/local/bin/s3cmd --stop-on-error --delete-removed sync s3://dkb-gitlab-gitlab-uploads-eu-central-1/ /srv/gitlab/tmp/uploads/
Problem: <class 'OSError: [Errno 36] File name too long: '/srv/gitlab/tmp/uploads/@hashed/71/ee/71ee45a3c0db9a9865f7313dd3372cf60dca6479d46261f3542eb9346e4a04d6/028acbe35bbe90de09dc21d3d4159916/.s3cmd.9lj_ex2h.tmp' -> b'/srv/gitlab/tmp/uploads/@hashed/71/ee/71ee45a3c0db9a9865f7313dd3372cf60dca6479d46261f3542eb9346e4a04d6/028acbe35bbe90de09dc21d3d4159916/url_sa_i_url_https_3A_2F_2Fwww.latintimes.com_2Fburn-notice-series-finale-spoilers-how-did-it-all-end-and-which-characters-survived-explosive-finale_psig_AOvVaw2d2_5UBy4dGKayGeDAmyhT_ust_1622228720698000_source_images_cd_vfe_ved_0CAIQjRxqFwoTCODh84vH6vACFQAAAAAdAAAAABAD'
S3cmd:   2.2.0
python:   3.8.9 (default, Apr  6 2021, 18:45:51) 
[GCC 8.3.0]
environment LANG=C.UTF-8

Traceback (most recent call last):
  File "/usr/local/bin/s3cmd", line 3209, in <module>
    rc = main()
  File "/usr/local/bin/s3cmd", line 3106, in main
    rc = cmd_func(args)
  File "/usr/local/bin/s3cmd", line 1967, in cmd_sync
    return cmd_sync_remote2local(args)
  File "/usr/local/bin/s3cmd", line 1533, in cmd_sync_remote2local
    ret, seq, size_transferred = _download(remote_list, seq, remote_count + update_count, size_transferred, dir_cache)
  File "/usr/local/bin/s3cmd", line 1387, in _download
    os.rename(chkptfname_b, deunicodise(dst_file))
OSError: [Errno 36] File name too long: '/srv/gitlab/tmp/uploads/@hashed/71/ee/71ee45a3c0db9a9865f7313dd3372cf60dca6479d46261f3542eb9346e4a04d6/028acbe35bbe90de09dc21d3d4159916/.s3cmd.9lj_ex2h.tmp' -> b'/srv/gitlab/tmp/uploads/@hashed/71/ee/71ee45a3c0db9a9865f7313dd3372cf60dca6479d46261f3542eb9346e4a04d6/028acbe35bbe90de09dc21d3d4159916/url_sa_i_url_https_3A_2F_2Fwww.latintimes.com_2Fburn-notice-series-finale-spoilers-how-did-it-all-end-and-which-characters-survived-explosive-finale_psig_AOvVaw2d2_5UBy4dGKayGeDAmyhT_ust_1622228720698000_source_images_cd_vfe_ved_0CAIQjRxqFwoTCODh84vH6vACFQAAAAAdAAAAABAD'

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  and have a look at the known issues list:
    https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions-(FAQ)
  If the error persists, please report the
  above lines (removing any private
  info as necessary) to:
   s3tools-bugs@lists.sourceforge.net

Results of GitLab environment info

System information
System:
Proxy:          GITLAB_GITLAB_PAGES_SERVICE_PORT_HTTP_GITLAB_PAGES_PROXY: 8090
Current User:   git
Using RVM:      no
Ruby Version:   2.7.5p203
Gem Version:    3.2.19
Bundler Version:2.2.19
Rake Version:   13.0.6
Redis Version:  unknown
Git Version:    unknown
Sidekiq Version:6.3.1
Go Version:     unknown

GitLab information
Version:        14.7.2-ee
Revision:       39a169b2f25
Directory:      /srv/gitlab
DB Adapter:     PostgreSQL
DB Version:     12.4
URL:            https://gitlab.dev.codefactory.sh
HTTP Clone URL: https://gitlab.dev.codefactory.sh/some-group/some-project.git
SSH Clone URL:  git@gitlab.dev.codefactory.sh:some-group/some-project.git
Elasticsearch:  yes
Geo:            no
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers: saml, saml_2

GitLab Shell
Version:        13.22.2
Repository storage paths:
- default:      /var/opt/gitlab/repo
GitLab Shell path:              /home/git/gitlab-shell
Git:            /usr/bin/git
git@gitlab-toolbox-backup-manual-z9fu3-vnt4x:/$

Possible fixes

Manual fix documented: !88872 (merged)