Feature validation: try migrating a whole shard of projects with the `repack_after_shard_migration` feature flag enabled
In gitlab-org/gitaly#2108 (closed) we introduced the repack_after_shard_migration
feature flag, defaulting to disabled.
A GC operation on a git repository can be I/O-unfriendly, and I have a doubt about the implementation - what happens when we migrate lots of repositories at the same time? I think this doubt can only be answered empirically.,
So, before removing the feature flag, I'd like to do a test, either in staging or production, of migrating all repositories on a shard to a new destination, using whatever procedures gitlab would normally use for an emergency evacuation. We can try with the feature flag disabled, and with it enabled, and watch metrics of shard health - especially base and peak IOPS - to determine whether the acute and chronic effects of this change are acceptable.
I don't think this is something I should trigger alone - it seems like something I should be doing with the aid of an SRE! Probably on staging first, and then production if we don't see any obvious red flags from the staging attempt.
cc @m_gill