## Summary The 12.4.0 `task-runner` images upgrade Python from 2.7 to 3.7, which breaks `s3cmd` in various situations. ## Steps to reproduce Run the `backup-utility` to backup or restore from AWS S3 with IAM credentials from instance metadata, Kiam or (presumably) Kube2iam. ## Configuration used ```yaml global: hosts: minio: enabled: false appConfig: lfs: bucket: git-lfs.gitlab.example.com artifacts: bucket: artifacts.gitlab.example.com uploads: bucket: uploads.gitlab.example.com packages: bucket: packages.gitlab.example.com backups: bucket: backups.gitlab.example.com tmpBucket: backups-temp.gitlab.example.com registry: bucket: registry.gitlab.example.com gitlab: task-runner: annotations: iam.amazonaws.com/role: GitLab-IAM-Role-Name backups: objectStorage: config: secret: gitlab-s3 key: s3cmd-config ``` This particular example uses Kiam, hence the annotation on the `task-runner` pods. Irrelevant config removed. ## Current behavior ``` Attempting to run '/bin/bash -c cp /etc/gitlab/.s3cfg $HOME/.s3cfg && backup-utility' as a main process WARNING: This version of GitLab depends on gitlab-shell 10.2.0, but you're running Unknown. Please update gitlab-shell. 2019-10-24 10:52:04 +0000 -- Dumping database ... Dumping PostgreSQL database gitlab ... [DONE] 2019-10-24 10:52:16 +0000 -- done WARNING: This version of GitLab depends on gitlab-shell 10.2.0, but you're running Unknown. Please update gitlab-shell. 2019-10-24 10:52:31 +0000 -- Dumping repositories ... [...] 2019-10-24 10:52:54 +0000 -- done Bucket not found: registry.gitlab.example.com. Skipping backup of registry ... Bucket not found: uploads.gitlab.example.com. Skipping backup of uploads ... Bucket not found: artifacts.gitlab.example.com. Skipping backup of artifacts ... Bucket not found: git-lfs.gitlab.example.com. Skipping backup of lfs ... Bucket not found: packages.gitlab.example.com. Skipping backup of packages ... WARNING: This version of GitLab depends on gitlab-shell 10.2.0, but you're running Unknown. Please update gitlab-shell. Packing up backup tar ERROR: /home/git/.s3cfg: None ERROR: Configuration file not available. ERROR: Consider using --configure parameter to create one. ``` The buckets to back up from are skipped, and the backup can't be uploaded to S3. ## Expected behavior Should be obvious! This worked before 12.4.0 though. ## Versions - Chart: `2.4.0` and `2.4.1` - Platform: - Cloud: EKS - Kubernetes: (`kubectl version`) - Client: `v1.16.2` - Server: `v1.14.6-eks-5047ed` - Helm: (`helm version`) - Client: `v2.14.3` - Server: `v2.14.3` ## Relevant logs See above.