[Feature flag] Roll out `:cleanup_lfs_during_gc`

What

Remove the :cleanup_lfs_during_gc feature flag ...

Owners

• Team: Create: Source Code
• Most appropriate slack channel to reach out to: #g_create_source-code-be
• Best individual to reach out to: @nick.thomas

Expectations

What are we expecting to happen?

Project housekeeping should now routinely remove unreferenced LFS objects

What might happen if this goes wrong?

The extra work during GitGarbageCollectWorker execution could be unwelcome

If it erroneously removes LFS objects when it shouldn't, we could lose data 😱

What can we monitor to detect problems with this?

The main concern I have is around performance. We're adding more work to an expensive worker, and this could cause problems.

We should watch:

• https://dashboards.gitlab.net/d/sidekiq-queue-detail/sidekiq-queue-detail?orgId=1&from=now-6h&to=now&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&var-queue=git_garbage_collect
• https://dashboards.gitlab.net/d/alerts-sat_disk_sus_read_iops/alerts-disk_sustained_read_iops-saturation-detail?orgId=1
• https://dashboards.gitlab.net/d/alerts-sat_disk_sus_write_iops/alerts-disk_sustained_write_iops-saturation-detail?orgId=1
• https://log.gprd.gitlab.net/goto/6618e5da43cfb62b9339db6f726a0fd2

Beta groups/projects

If applicable, any groups/projects that are happy to have this feature turned on early. Some organizations may wish to test big changes they are interested in with a small subset of users ahead of time for example.

• nick.thomas/www-gitlab-com project
• nick.thomas/gitlab-ee project
• gitlab-org group
• gitlab-com group

Roll Out Steps

• Enable on staging
• Test on staging
• Ensure that documentation has been updated !40979 (merged)
• Enable on GitLab.com for individual groups/projects listed above and verify behaviour
• Coordinate a time to enable the flag with #production and #g_delivery on slack.
• Announce on the issue an estimated time this will be enabled on GitLab.com
• Enable on GitLab.com by running chatops command in #production
• Cross post chatops slack command to #support_gitlab-com (more guidance when this is necessary in the dev docs) and in your team channel
• Announce on the issue that the flag has been enabled
• Remove feature flag and add changelog entry
• After the flag removal is deployed, clean up the feature flag by running chatops command in #production channel