Delete old unused artifacts
Problem to solve
Marcel brought this up in Slack:
I just had a meeting with a pipeline insights engineer and we were talking about job artifacts and some problems with deleting them... and I just checked the docs project to see how many we have. We have 114 gigabytes of job artifacts: https://gitlab.com/gitlab-org/gitlab-docs/-/usage_quotas
That seems... high... Do we have any processes that need to make use of older job artifacts, or do you think it'd be safe to use this API to try to bring down the usage? https://docs.gitlab.com/ee/api/job_artifacts.html#delete-project-artifacts
It'd be interesting to check and see how much it actually deletes, and how much it misses (likely due to the "artifacts from the most recent successful pipelines are always kept" feature.
I checked and we expire the build site artifacts after one day:
I verified that this is the case by going to an old pipeline and checking the artifacts available to download.
Proposal
Let's use this issue to
-
Disable the Keep artifacts from most recent successful jobs feature so that all artifacts can expire when they are no longer needed. -
Fix our CI/CD yaml files to expire the security jobs artifacts after 1h
(and possibly do the same for the build site artifacts). -
Delete old artifacts using the API