Make Object Storage a first class citizen in GitLab

Description

We are working to add the ability to store artifacts in object storage (https://gitlab.com/gitlab-org/gitlab-ce/issues/29203), and we have plans to be able to support other types of data there like Git LFS.

For larger deployments this is going to be a popular feature, as it allows linear scalability for most types of storage. However for existing customers, it could be challenging to actually embrace this. The primary reason is that we currently offer no method of migration and do not support tracking which assets are stored where. We instead rely on the server administrator to take this action to move the data themselves, which is a significant undertaking.

We should provide a simple checkbox to enable this feature, and do the right thing for our customers.

Proposal

We should offer the ability to configure S3 storage for artifacts in the GitLab administration UI. We should expose the necessary S3 configuration data, and confirm connectivity and valid credentials. Enabling this would cause all net-new artifacts to be stored in S3. Existing artifacts would remain stored locally, and still be available. We should be able to simply track where the underlying file is, and have a simple flag for whether it's on S3 or local disk.

Then, we should have another option on whether we should migrate the data over to S3. This would begin a background copy job, slowly copying artifacts to S3. As they are copied, we would then update the location flag noted above to indicate they are in S3. After the copy is complete, we should notify the administrator and they can backup or simply delete the local artifacts.

By making moving to object storage easy, it would be a more compelling value for our CE users with large installations.