Clean up old artifacts
Problem
This repo is using ~130 gigabytes just for job artifacts. This is likely due to two issues:
-
Our artifacts are really quite large. This is not something we can change right now.
-
Old artifacts are not being deleted. This is something we can deal with:
- In 13.0, a new feature was added that forced all "latest" artifacts to not be deleted: https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#keep-artifacts-from-most-recent-successful-jobs. This was enabled by default and could not be changed. Later, a setting was added in 13.9 where you could disable the feature, but we didn't realize we needed to disable it. As a result, a large number of old pipelines (like for stable branches) still have their artifacts taking up space, despite our pipeline config set to delete them within 24 hours.
- For example:
- Pipeline for 12.9: https://gitlab.com/gitlab-org/gitlab-docs/-/jobs/605333460
The artifacts were removed 2 years ago
- Pipeline for 13.2: https://gitlab.com/gitlab-org/gitlab-docs/-/jobs/767721557
These artifacts are the latest. They will not be deleted (even if expired) until newer artifacts are available.
(~600M, though slowly crept up until ~1G in more recent pipelines).
- Pipeline for 12.9: https://gitlab.com/gitlab-org/gitlab-docs/-/jobs/605333460
I have since disabled the "keep artifacts" feature, but turning it off does not release older artifacts for expiry. We need to either:
- Run the old pipelines again to try generating a new artifact, which can then be allowed to expire "naturally". No idea if it'll work, and the last job in a lot of these pipelines is a deployment, I'm not sure I want to test the robustness of those old pipelines.
- Use this API endpoint to tag all artifacts as ready for deletion. <- My proposal
I think we should use the API and just do a blanket delete, because we do not use these old artifacts in any process. After a pipeline completes, there is no more need to keep them. In fact, we set them to delete in just 1 day. For example:
- Pipeline from yesterday: https://gitlab.com/gitlab-org/gitlab-docs/-/jobs/4005654618
The artifacts were removed 58 minutes ago
Edited by Marcel Amirault