s3cmd with S3 and instance credentials broken with 12.4.0

Summary

The 12.4.0 task-runner images upgrade Python from 2.7 to 3.7, which breaks s3cmd in various situations.

Steps to reproduce

Run the backup-utility to backup or restore from AWS S3 with IAM credentials from instance metadata, Kiam or (presumably) Kube2iam.

Configuration used

global:
  hosts:
  minio:
    enabled: false

  appConfig:
    lfs:
      bucket: git-lfs.gitlab.example.com

    artifacts:
      bucket: artifacts.gitlab.example.com

    uploads:
      bucket: uploads.gitlab.example.com

    packages:
      bucket: packages.gitlab.example.com

    backups:
      bucket: backups.gitlab.example.com
      tmpBucket: backups-temp.gitlab.example.com

  registry:
    bucket: registry.gitlab.example.com

gitlab:
  task-runner:
    annotations:
      iam.amazonaws.com/role: GitLab-IAM-Role-Name

    backups:
      objectStorage:
        config:
          secret: gitlab-s3
          key: s3cmd-config

This particular example uses Kiam, hence the annotation on the task-runner pods. Irrelevant config removed.

Current behavior

Attempting to run '/bin/bash -c cp /etc/gitlab/.s3cfg $HOME/.s3cfg && backup-utility' as a main process
WARNING: This version of GitLab depends on gitlab-shell 10.2.0, but you're running Unknown. Please update gitlab-shell.
2019-10-24 10:52:04 +0000 -- Dumping database ... 
Dumping PostgreSQL database gitlab ... [DONE]
2019-10-24 10:52:16 +0000 -- done
WARNING: This version of GitLab depends on gitlab-shell 10.2.0, but you're running Unknown. Please update gitlab-shell.
2019-10-24 10:52:31 +0000 -- Dumping repositories ...
[...]
2019-10-24 10:52:54 +0000 -- done
Bucket not found: registry.gitlab.example.com. Skipping backup of registry ...
Bucket not found: uploads.gitlab.example.com. Skipping backup of uploads ...
Bucket not found: artifacts.gitlab.example.com. Skipping backup of artifacts ...
Bucket not found: git-lfs.gitlab.example.com. Skipping backup of lfs ...
Bucket not found: packages.gitlab.example.com. Skipping backup of packages ...
WARNING: This version of GitLab depends on gitlab-shell 10.2.0, but you're running Unknown. Please update gitlab-shell.
Packing up backup tar
ERROR: /home/git/.s3cfg: None
ERROR: Configuration file not available.
ERROR: Consider using --configure parameter to create one.

The buckets to back up from are skipped, and the backup can't be uploaded to S3.

Expected behavior

Should be obvious! This worked before 12.4.0 though.

Versions

• Chart: 2.4.0 and 2.4.1
• Platform:
  • Cloud: EKS
• Kubernetes: (kubectl version)
  • Client: v1.16.2
  • Server: v1.14.6-eks-5047ed
• Helm: (helm version)
  • Client: v2.14.3
  • Server: v2.14.3

Relevant logs

See above.