Database backups in EU bucket are not cleaned up
We replicate database backups from the US bucket to the EU bucket in S3 using cross-region replication. As pointed out in https://gitlab.com/gitlab-com/infrastructure/issues/4365#note_81124252, versioning is in place to prevent malicious deletes being replicated to the EU bucket.
Consequently, it looks like the EU bucket is never cleaned up at all. In fact, we can still find basebackups and corresponding WAL back until May 2017 in postgres-01
and postgres-02
"folder". Those appear to be deleted but their earlier versions are still present and likewise we're getting billed for the space.
Proposal here is to add a lifecycle rule to the EU bucket that removes any objects older than 30 days. This is a 3 weeks longer than the default retention interval we have configured for wal-e (the wal-e trim
cronjob only retains the latest 8 days worth of backups in the US bucket).