## Summary - **DRI**: @sgarg_gitlab - **Slack channel**: `#<TODO: team slack channel>` - **Feature issue**: https://gitlab.com/gitlab-org/gitlab/-/issues/602881 - **Introduced by**: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/251124 CarrierWave can leave `Upload` rows behind when a model with a mounted upload is destroyed, if the file is already missing from storage or the mounted column is empty. This flag gates a `before_destroy` callback, `sweep_mounted_uploads`, added to the shared `WithUploads` concern. It runs while the owning model still exists, so Geo can locate the file, and it queues the row deletion and file cleanup for after commit. When CarrierWave has already deleted things correctly, the sweep finds nothing and returns early. The flag is **instance-scoped**, not actor-scoped, because 13 files include `WithUploads` and there is no actor type common to all of them (`AbuseReport`, `Appearance`, and `BulkImports::ExportUpload` cannot be passed as an actor without raising). This means the rollout is a plain on/off switch. There is no percentage-based rollout for this flag. ## What could go wrong? - **Blast radius**: the callback applies to every model that includes `WithUploads`. That is 13 files: 12 models (`AbuseReport`, `Appearance`, `BulkImports::ExportUpload`, `DesignManagement::Action`, `Group`, `ImportExportUpload`, `Organizations::OrganizationDetail`, `PersonalSnippet`, `Project`, `Projects::ImportExport::RelationExportUpload`, `User`, `UserPermissionExportUpload` in EE) plus the `MetricImageUploading` concern, which brings in `AlertManagement::MetricImage` and `IssuableMetricImage`. 14 models in total. - **Extra query cost on every destroy**: `begin_fast_destroy` and `delete_all` both run unconditionally, even when there is nothing to clean up. `Project`, `Group`, and `User` destroys are the highest-traffic of these paths, so that is where the added cost is most visible. - **Irreversible if the scope is wrong**: when the sweep does find rows it runs `delete_all` on `uploads` and schedules async file deletion via `DeleteStoredFilesWorker`, plus Geo deletion events. A wrong scope would delete files that are still in use, and that is not recoverable. The scope is `model_type` + `id` + the model's own mounted uploader class names, so it cannot match another model's rows. - **Rollback is clean**: the sweep only removes rows CarrierWave already failed to delete. Disabling the flag returns to current `master` behaviour; nothing is left half-done. - **What to watch**: Geo verification failure metrics (should go down), `uploads` table query volume, and `DeleteStoredFilesWorker` job volume (should stay near zero unless orphans actually exist). ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to the team channel. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process). This flag is instance-scoped, so the `--actors` percentage commands do not apply. It is enabled or disabled outright. **Non-production** ``` /chatops gitlab run feature set sweep_orphaned_mounted_uploads true --dev --pre --staging --staging-ref ``` **Production** ``` /chatops gitlab run feature set sweep_orphaned_mounted_uploads true ``` After enabling, confirm on the Geo primary that newly destroyed models leave no `Upload` rows behind, and that `DeleteStoredFilesWorker` volume stays proportionate to the orphans actually found. ## Before global rollout - [x] Verified on staging that destroying a model with a missing upload file removes the `Upload` row - [x] `uploads` query volume and destroy-path latency checked on https://dashboards.gitlab.net - [ ] Geo verification failure count for uploads trending down - [ ] [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required Documentation and version history do not apply: `gitlab_com_derisk` flags are short-lived and are not listed in the all-feature-flags page. ## Cleanup Remove the flag in milestone 19.5. GitLab Dedicated does not support feature flags, and issue 602881 carries the `GitLab Dedicated` label, so the fix does not reach Dedicated until the flag is removed and the behaviour is default-on. `gitlab_com_derisk` flags have a maximum lifespan of two months after merge, which bounds this. Remove the flag check in `app/models/concerns/with_uploads.rb`, its YAML definition in `config/feature_flags/gitlab_com_derisk/sweep_orphaned_mounted_uploads.yml`, and the disabled-state spec context in `spec/models/concerns/with_uploads_spec.rb`. Then: ``` /chatops gitlab run release check <merge-request-url> 19.5 /chatops gitlab run feature delete sweep_orphaned_mounted_uploads --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set sweep_orphaned_mounted_uploads false # production /chatops gitlab run feature set sweep_orphaned_mounted_uploads false --dev --pre --staging --staging-ref # non-production ```