Improvements to the storage_migration.rb script
During: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6147
A few improvements have been proposed:
- Auto log to a file without the need for
tee - Dynamic wait time
- Better Error handling
- One example, we query all projects at the start, and then query each project id later, if the project doesn't exist that second time, the script bails
- Monitor for alerts for each file server being operated on to adjust whether more work should be queued
- Detect job failures so we aren't waiting unnecessarily