Skip to content

Detect and fix artifacts with backfilled expire_at

Albert Salim requested to merge 353128-resume-clean-up-expired-artifact into master

What does this MR do and why?

This detects and fixes job artifacts that have expire_at wrongly backfilled by the migration !47723 (merged). These job artifacts will not be deleted and will have their expire_at removed.

The migration would have backfilled expire_at to midnight on the 22nd of the month of the local timezone in UTC time. If the local timezone since the migration, the expire_at stored in the database could have changed to a different local time other than midnight. For example:

  • changing timezone from UTC+02:00 to UTC+02:30 would change the expire_at in local time 00:00:00 to 00:30:00.
  • changing timezone from UTC+00:00 to UTC-01:00 would change the expire_at in local time 00:00:00 to 23:00:00 on the previous day (21st).

Therefore job artifacts that have expire_at exactly on the 00, 30 or 45 minute mark on the dates 21, 22, 23 of the month will not be deleted. https://en.wikipedia.org/wiki/List_of_UTC_time_offsets

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

  1. Enable the feature flag ci_destroy_all_expired_service and ci_detect_wrongly_expired_artifacts.
  2. Create some job artifacts.
  3. In rails console, overwrite some job artifacts' expire_at to be the 22 of the month at midnight.
  4. Expire the job artifacts.
  5. Ensure all workers involved in deleting job artifacts from storage have completed.
  6. Verify in object storage that the job artifact files has been removed, except those that expire at midnight on 22nd of the month.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #353128 (closed)

Edited by Albert Salim

Merge request reports