Discussion: migration methods for future filesystem -> object store
We have had a lot of good discussion on how to migrate files from the filesystem onto an object store. In particular, we have had some concerns about the way we are migrating artifacts to S3. For context on this, please see #2807, #2772, and #2387 (closed).
We need to create a solution that will work for everyone, including us. GitLab.com is a very special case as we host several TB of artifacts, uploads, and build logs. Generally, a self-hosted instance will not have nearly this kind of data.
For artifacts, we decided to run a script in the rails console and check in on it from time to time. It was suggested that we use background migrations in the future as we could have more debugging and monitoring via that method.
We need to come up with a solution on how we intend to process future migrations as we will definitely have them. Do we use background migrations, a rails console command, rake task, or something entirely different? What is possible and how long would it take to implement the various different methods of migration? What other requirements have I forgotten?
cc/ @gl-infra @ayufan @zj @yorickpeterse