s3cmd breakage in 12.4.0 with external Minio

Summary

s3cmd fails to sync from a Minio bucket with task-runner 12.4.0.

Similar to #1649 (closed), python-related issues in s3cmd are affecting backups.

Steps to reproduce

This GitLab instance is using an external Minio instance (not the one supplied with the GitLab chart). Just run the backup-utility.

Configuration used

The chart config:

global:
  minio:
    enabled: false

  appConfig:
    lfs:
      bucket: lfs.gitlab.example.com

    artifacts:
      bucket: artifacts.gitlab.example.com

    uploads:
      bucket: uploads.gitlab.example.com

    packages:
      bucket: packages.gitlab.example.com

    backups:
      bucket: backups.gitlab.example.com
      tmpBucket: backups-tmp.gitlab.example.com

  registry:
    bucket: registry.gitlab.example.com

gitlab:
  task-runner:
    backups:
      objectStorage:
        config:
          secret: gitlab-minio
          key: s3cmd-config

The s3cmd-config config from the gitlab-minio Secret:

[default]
access_key = SNIPPED
secret_key = SNIPPED
bucket_location = us-east-1
enable_multipart = True
host_base = minio.example.com
host_bucket = minio.example.com/%(bucket)
multipart_chunk_size_mb = 100
multipart_max_chunks = 1000
preserve = False
check_md5 = False
website_endpoint = https://minio.example.com

Current behavior

Begin parsing .erb files from /var/opt/gitlab/templates
Writing /srv/gitlab/config/resque.yml
Writing /srv/gitlab/config/gitlab.yml
Writing /srv/gitlab/config/database.yml
Copying other config files found in /var/opt/gitlab/templates
Copying smtp_settings.rb into /srv/gitlab/config
Attempting to run '/bin/bash -c cp /etc/gitlab/.s3cfg $HOME/.s3cfg && backup-utility' as a main process
WARNING: This version of GitLab depends on gitlab-shell 10.2.0, but you're running Unknown. Please update gitlab-shell.
2019-10-24 11:38:29 +0000 -- Dumping database ... 
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2019-10-24 11:38:34 +0000 -- done
WARNING: This version of GitLab depends on gitlab-shell 10.2.0, but you're running Unknown. Please update gitlab-shell.
2019-10-24 11:38:55 +0000 -- Dumping repositories ...
[...]
Dumping registry ...
Restore registry failed
[Error] 
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  and have a look at the known issues list:
    https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions
  If the error persists, please report the
  following lines (removing any private
  info as necessary) to:
   s3tools-bugs@lists.sourceforge.net


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Invoked as: /usr/local/bin/s3cmd --stop-on-error --delete-removed sync s3://registry.gitlab.example.com/ /srv/gitlab/tmp/registry/
Problem: <class 'TypeError: %d format: a number is required, not NoneType
S3cmd:   2.0.1
python:   3.7.3 (default, Sep 30 2019, 15:48:19) 
[GCC 6.3.0 20170516]
environment LANG=C.UTF-8

Traceback (most recent call last):
  File "/usr/local/bin/s3cmd", line 1316, in _download
    response = s3.object_get(uri, dst_stream, dst_file, extra_label = seq_label)
  File "/usr/local/lib/python3.7/site-packages/S3/S3.py", line 715, in object_get
    response = self.recv_file(request, stream, labels, start_position)
  File "/usr/local/lib/python3.7/site-packages/S3/S3.py", line 1612, in recv_file
    http_response = conn.c.getresponse()
  File "/usr/local/lib/python3.7/http/client.py", line 1321, in getresponse
    response.begin()
  File "/usr/local/lib/python3.7/site-packages/S3/Custom_httplib3x.py", line 53, in httpresponse_patched_begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.7/http/client.py", line 265, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/s3cmd", line 3073, in <module>
    rc = main()
  File "/usr/local/bin/s3cmd", line 2989, in main
    rc = cmd_func(args)
  File "/usr/local/bin/s3cmd", line 1875, in cmd_sync
    return cmd_sync_remote2local(args)
  File "/usr/local/bin/s3cmd", line 1470, in cmd_sync_remote2local
    ret, seq, size_transferred = _download(remote_list, seq, remote_count + update_count, size_transferred, dir_cache)
  File "/usr/local/bin/s3cmd", line 1347, in _download
    error(u"Download of '%s' failed (Reason: Unknown OsError %d)" % (file, exc.errno))
TypeError: %d format: a number is required, not NoneType

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    An unexpected error has occurred.
  Please try reproducing the error using
  the latest s3cmd code from the git master
  branch found at:
    https://github.com/s3tools/s3cmd
  and have a look at the known issues list:
    https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions
  If the error persists, please report the
  above lines (removing any private
  info as necessary) to:
   s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Expected behavior

The backups should work, as they did pre-12.4.0.

Versions

  • Chart: 2.4.0 and 2.4.1
  • Platform:
    • Self-hosted: Rancher RKE
  • Kubernetes: (kubectl version)
    • Client: v1.16.2
    • Server: v1.15.5
  • Helm: (helm version)
    • Client: v2.14.3
    • Server: v2.14.3

Relevant logs

See above.

Edited by Jason Plum