Backup utility does not remove non existing items while syncing from remote bucket on GCP
Summary
Backup cronjob uses a PVC to store data coming from syncing remote buckets, dumping DB, packing everything into a tarball, and pushing it to the backup bucket.
In the case of GCP, the usage of gsutil rsync
without -d
flag makes content that is present on the local copy of the bucket but not on the remote one, to be present in the final final version backup tarball file.
-d Delete extra files under dst_url not found under src_url. By
default extra files are not deleted. Note: this option can
delete data quickly if you specify the wrong source/destination
combination. See the help section above,
"BE CAREFUL WHEN USING -d OPTION!".
Steps to reproduce
- Create a pipeline that pushes artifacts with 1h availability, and execute it several times.
- Make an initial backup from and to GCP
- Check the content of the backup if artifacts pushed are present
- Wait for a couple of hours
- The artifacts on the bucket should not be there anymore as they are expired
- Execute the backup process again
- Check the content of the backup if artifacts matches the bucket content
Configuration used
Not relevant
Current behavior
gsutil rsync
is keeping files in the local directory that are not present in the bucket
Expected behavior
gsutil rsync
with -d
flag should remove all files present in the local directory that are not present in the bucket
Versions
- Chart: 5.9.1
- Platform:
- Cloud: GKE
- Kubernetes: (
kubectl version
)- Client:
- Server:
- Helm: (
helm version
)- Client:
- Server:
Relevant logs
Edited by Mitchell Nielsen