# TL;DR The migration added in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47723+, which was written to satisfy a `gitlab.com` requirement[^1], will eventually cause silent, **unrecoverable data loss** for all self-hosted instances. :exclamation: [^1]: See #263234 which starts "To support the change at https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10177 ..." Since then, we've introduced the `ci_destroy_all_expired_service` feature flag in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/76504/ which has turned the removal of expired artifacts off by default for self-hosted customers to prevent any further data loss. We will need to ship a migration https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78806 aimed specifically at fixing artifact expiration for self-hosted customers before we turn the flag back on by default. In the mean time, any customer can opt back in to normal expiration by turning `ci_destroy_all_expired_service` back on if they know they aren't worried about removing old artifacts or job traces. # Problems 1. The migration `backfill_artifact_expiry_date.rb`, added in [!47723](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47723) retroactively sets an expiration date on ***all*** "old" artifacts which do not have an expiration date. This includes artifacts *for tagged builds* which many workflows deem to be precious. **This was never mentioned on the [GitLab 13.8 release blogpost](https://about.gitlab.com/releases/2021/01/22/gitlab-13-8-released/)!** 2. The only reason that most users have not yet seen this problem is because the service which actually destroys expired artifacts is broken and doesn't work when it encounters large "slugs" of artifacts all of which have the same expiration date: [#330378 `DestroyAllExpiredService` calls `each_batch` in an unsafe manner](https://gitlab.com/gitlab-org/gitlab/-/issues/330378) Once [#330378](https://gitlab.com/gitlab-org/gitlab/-/issues/330378) is fixed, self-hosted admins will start getting complaints from their users that their old artifacts are inexplicably deleted. :boom: 3. The migration does not filter on `file_type`; it even deletes [job logs](https://docs.gitlab.com/ee/user/gitlab_com/index.html#job-logs), which are normally excluded from artifact expiration. Adding to the frustration, because deleted artifacts get their rows deleted from the database, without database backups there is no way to determine which files or rows have been removed. 4. The migration wasn't even implemented correctly and had to be backed-out and re-scheduled: - @stanhu pointed out that it was failing when it was pushed to production https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47723/diffs#note_486021780 - [!51821](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/51821), [!51822](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/51822), [!55093](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/55093) 5. While this migration set `ci_job_artifacts.expire_at`, it did *not* set `ci_builds.artifacts_expire_at` (which is terribly redundant). The result is that [this code](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.9.0-ee/app/serializers/build_details_entity.rb#L29-53): 1. Does not tell you that the artifacts will expire if the current time is *before* the artifactopocalypse 2. Does not let you <kbd>Keep</kbd> artifacts 3. Does not tell you when artifacts were removed if the current time is *after* artifactopocalypse (whether or not `DestroyAllExpiredService` *actually* was able to remove them) (Added based on [this comment](https://gitlab.com/gitlab-org/gitlab/-/issues/343302#note_708276462) and gitlab 13.9.5) ## Proposal 1. Fix Job Log expiration - SaaS focused: revert `expire_at` to `nil` where `it is set to the 22nd of any month at midnight local time` and `file_type` is one of the non-expiring artifacts (e.g. trace). 1. Fix Job Log expiration - Self-hosted focused: revert `expire_at` to `nil` where `it is set to the 22nd of any month at midnight local time` regardless of the `file_type`. ```sql update ci_job_artifacts set expire_at = null where extract(day from expire_at) = 22 and (expire_at at time zone 'US/Eastern')::time = '00:00:00'; ``` 1. Revert the change for self-hosted only as Fabio [describes here](https://gitlab.com/gitlab-org/gitlab/-/issues/343302#note_708862194), we can repurpose https://gitlab.com/gitlab-org/gitlab/-/issues/343372 for that I'd think as the intent is the same, undo the migration. * I'm unsure if we also need to write a migration to change the expire_at values or if this "revert" would be that migration. * Write a migration to change expire_at values that were set after upgrade to 13.8 was performance so `expire_at` is set to NULL if it is set to the 22nd of any month at midnight local time 1. Deprecate `ci_builds.artifacts_expire_at` and ensure all code is refactored to reference `ci_job_artifacts.artifacts_expire_at` and `ci_pipeline_artifacts.artifacts_expire_at` (cc @morefice i'm making an assumption we used a similar pattern for pipeline artifacts) 1. Implement #19583 for better user experience in wanting to keep specific artifacts in the UI * We would also want to ensure the [Job Artifacts API](https://docs.gitlab.com/ee/api/job_artifacts.html#keep-artifacts) uses this as well. * @morefice @shampton We don't offer a UI for pipeline artifacts today but if that gets created at some point we'd want to refactor to use this new column.