`@final` artifacts not deleted from object store
We run a private Gitlab CE instance and after the upgrade to 16.1.0 we noticed that our S3 bucket used for CI job artifacts continued to grow rather quickly.
We set a default artifact timeout of 2 weeks, but even after a month, the storage requirement keeps growing, and I see lots of files with an @final
subdirectory in their path that were created right after we installed 16.1.0. When I check in the database, I see that select sum(size) from ci_job_artifacts;
shows about 5TB usage, while on S3 I see around 18TB and growing. In the ci_deleted_objects
table I see between 0 and a few hundred entries that appear to be regularly cleaned up by the cron job as expected.
I believe those files I see were added by 426ecb7e which was enabled by 9c18ebe6 in 16.1.0 by default. Could it be, that the new paths are missing in some cleanup job and thus the artifacts are deleted from the database, but not from the storage?
I also do see that some space is freed regularly, but not all. Could it be that artifacts are saved in two locations and only one is freed? While writing this I figured I could just compare directory lists of the S3 bucket, but that will take some more time, until files are deleted. I'll update this report (possibly tomorrow) with that additional information. Edit: This might be an artifact from another user/bucket on the same account since I only have aggregated metrics. It looks like just @final
files get added and never removed again.
We currently have the following feature flags active:
- multiple_merge_request_assignees
- validate_import_decompressed_archive_size
- ci_destroy_all_expired_service
- ci_artifact_fast_removal_large_loop_limit
- ci_job_artifacts_backlog_work
- ci_job_artifacts_backlog_large_loop_limit
Our current version is 16.2.0. I see that there is a 16.2.1, but !127401 (merged) does not sound like it would fix this issue.