Task Runner cron job fails to backup.
Cross-posting from main gitlab issues.
https://gitlab.com/gitlab-org/gitlab-ce/issues/65461#note_199694553
Hello
I have deployed gitlab via helm chart and everything seems to work except the backup job. The cronjob fails and boto complains about some errors.
root@LDMKBQF2:/mnt/c/Git/ops-infra/gke-config/helm# helm ls
NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
cert-manager 1 Fri Jul 26 15:49:00 2019 DEPLOYED cert-manager-v0.8.1 v0.8.1 cert-manager
gitlab 12 Thu Aug 1 10:26:40 2019 DEPLOYED gitlab-2.1.3 12.1.3 default
kubed 1 Fri Jul 26 15:49:29 2019 DEPLOYED kubed-0.10.0 0.10.0 kube-system
nginx-ingress 4 Fri Jul 26 23:20:35 2019 DEPLOYED nginx-ingress-1.11.3 0.25.0 default
Relevant snippet of helm values.
gitlab:
task-runner:
backups:
cron:
enabled: true
schedule: 0 * * * *
objectStorage:
backend: gcs
config:
gcpProject: REDACTED
secret: google-credentials
key: creds.json
Helm version:
cy@cloudshell:~ (REDACTED)$ helm version
Client: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.2", GitCommit:"a8b13cc5ab6a7dbef0a58f5061bcc7c0c61598e7", GitTreeState:"clean"}
PS: Client PC is running v2.14.2. I didn't have access to that laptop when i filed this issue.
Kubernetes version:
cy@cloudshell:~ (REDACTED)$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.7", GitCommit:"4683545293d792934a7a7e12f2cc47d20b2dd01b", GitTreeState:"clean", BuildDate:"2019-06-06T01:46:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.7-gke.8", GitCommit:"7d3d6f113e933ed1b44b78dff4baf649258415e5", GitTreeState:"clean", BuildDate:"2019-06-19T16:37:16Z", GoVersion:"go1.11.5b4", Compiler:"gc", Platform:"linux/amd64"}
This is the stackdriver log entry.
I looked at the expanded helm chart and this is what it shows.
configure-gsutil: |
# The following script is used to configure gsutil when creating backups
# It provides inputs to the `gsutil config -e` prompt as follows:
# 1) Path to service account JSON key file
# 2) Do not set permissions for key file
# 3) GCP Project ID
# 4) Decline anonymous usage statistics
printf "$GOOGLE_APPLICATION_CREDENTIALS\nN\nREDACTED\nN\n" | gsutil config -e && while sleep 3600; do :; done
Finally, I can exec in to the task-runner pod and run the backup-utility successfully.
Any ideas what might be causing the job to fail?
Thank you.
Edited by Mahamed