"tmp/builds/BUILD_ID/chunks/0.log does not exist" error when backing up artifacts
Summary
The backup process could fail with the error related to the job log chunks when backing up artifacts.
Steps to reproduce
Exact steps are not clear as I was not able to reproduce it on a test instance:
- have a helm chart instance with
ci_enable_live_trace
feature flag enabled - run the GitLab backup including artifacts while there are some CI pipelines running
Configuration used
GCS is used as a backend for backups & object storage.
Current behavior
The backup process could intermittently fail with the errors like NotFoundException: 404 gs://gitlab1-artifacts/tmp/builds/380554/chunks/0.log does not exist.
when backing up artifacts.
The chunk files are created because New incremental logging architecture is enabled. I suppose the issue is happening because of the race condition: the pipelines are executed and the chunks are removed while the backup is in progress.
Expected behavior
The backup should be completed successfully. Maybe we need something similar to copy strategy for the backup script used in helm charts.
Versions
- Chart: gitlab-4.6.1
- Platform:
- Cloud: GKE
- Kubernetes:
- Client: v1.16.2
- Server: v1.16.15-gke.6000
- Helm:
- Client: 2.16.1
- Server: 2.16.1
Relevant logs
NotFoundException: 404 gs://gitlab1-artifacts/tmp/builds/380557/chunks/0.log does not exist.
NotFoundException: 404 gs://gitlab1-artifacts/tmp/builds/380555/chunks/0.log does not exist.
NotFoundException: 404 gs://gitlab1-artifacts/tmp/builds/380573/chunks/0.log does not exist.
...
CommandException: 16 files/objects could not be copied/removed.
/ [329/345 files][ 16.5 MiB/ 17.2 MiB] 96% Done 1.2 MiB/s ETA 00:00:01
command terminated with exit code 1
...
Possible workarounds
- skip artifacts when creating a backup
- create a backup when there are no pipelines running
- temporarily disable
ci_enable_live_trace