Backend: Introduce an API to delete all job artifacts before a given date
To alleviate many of the pain points related to job artifacts that stick around for a long time, we could introduce an API that deletes all job artifacts in a project before a given timestamp.
Current situation:
- There is an API to delete project artifacts
DELETE /projects/:id/artifacts
: https://docs.gitlab.com/ee/api/job_artifacts.html#delete-project-artifacts. However, this API only immediately deletes job artifacts that have been unlocked but not yet deleted because it has not passed theexpire_at
time. In addition, it does not delete job logs (trace). - There are multiple bugs currently that prevents a proper unlocking of job artifacts. This adds to the problem because these unlocked job artifacts cannot be deleted by
DELETE /projects/:id/artifacts
API.
Proposal
We could add the following parameters to the existing API DELETE /projects/:id/artifacts
:
-
ignore_lock
: Deletes all job artifacts regardless of lock status. It is a very forceful and destructive operation and cannot be recovered. The documentation must highlight this. -
created_before
: Deletes all job artifacts created before the given date. -
type
: Deletes job artifacts of the given type. This can be an array of types, orall
for everything.
To retain the existing behaviour, the parameters should have the following default values:
ignore_lock: false
created_before: nil
type: everything except logs
The benefit of implementing this as an API is that we can delegate the job artifact retention mechanism to each project, where each project can set up its own schedule to delete the job artifacts with parameters that suit the project's retention needs. This indirectly helps to address the job log retention issue as well.
The same API can be used for multiple purposes, for example:
- To delete job logs that are older than 90 days, run a daily job that calls the API with
ignore_lock: true, created_before: "date 90 days ago", type: trace
- To delete job artifacts that are unlocked but have not expired (current API behaviour), call the API without any parameters.
- To delete all job artifacts in existence, call the API with
ignore_lock: true, type: all
.