Update Clean Cache CI jobs for Deployer to clean out directories where debian packages are downloaded
Problem
Last week we ran into 2 issues where disks ran full. This commonly occurs when a deploy fails for any reason and our clean up tasks/jobs do not run. We've also recently switched how we download and install packages, using a direct GCS download vs. apt
, thus we are downloading packages in new location for which our apt cache clean ansible playbooks do not address. This new install method deletes packages if the install completes successfully, but if the install fails prior to the package deletion, we could build up a set of packages on our servers that are never cleaned up.
Reference
- Cache clean is defined in CI: https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/blob/7ed41eb53544180efb38e2b22dc885353bf70f1f/.gitlab-ci.yml#L117
- This is the playbook: https://ops.gitlab.net/gitlab-com/gl-infra/deploy-tooling/-/blob/master/clean-cache.yml
- New deletion method: https://ops.gitlab.net/gitlab-com/gl-infra/deploy-tooling/-/blob/5680bccd5aad9c52c8ff914634bf6b8a5df398f6/common_tasks/install_gitlab_ee.yml#L53
Solution
Adjust our cache cleaning jobs to ensure we clean up the temporary directory where packages are held locally.
Edited by John Skarbek