Migrate artifacts to Object Storage(S3)
This issue describes steps to get to the point of migrating artifacts to object storage.
The migration is online, and not interrupting to existing users.
Configure
Make GitLab be aware of Object Storage as described in https://docs.gitlab.com/ee/administration/job_artifacts.html#using-object-storage,
How it works?
GitLab stores information where artifacts are stored: local or remote storage. By default all artifacts are stored locally, enabling object storage doesn't change that, artifacts are continued to be stored locally, but we gain an ability to migrate all existing artifacts to object storage and be served from there.
How to migrate artifacts:
Migrate all artifacts starting from oldest one, this is currently synchronous, single threaded process:
gitlab-rake gitlab:artifacts:migrate
Migrate specific artifact for the specific build (from rails/console
):
Ci::Build.find(BUILD_ID).tap do |build|
build.artifacts_file.migrate!(ArtifactUploader::REMOTE_STORE)
build.artifacts_metadata.migrate!(ArtifactUploader::REMOTE_STORE)
end
Plan for making a change
- Decide where we want to store artifacts, S3, Google (S3-compatible)?
- Enable Object Storage in GitLab,
- Migrate single build and tests everything for that project,
- Migrate all artifacts for single project,
- Migrate all artifacts for all projects,
What we may need to do?
- We may need to improve the migration script to use threading approach to migrate multiple files at once,
- For 10.0 implement an ability to automatically upload all new artifacts once uploaded,
- For 10.1/2/3 make runner talk directly to object storage and reduce the egress traffic,
Who is gonna be involved from CI/CD team?
Initially @tmaczukin, later @ayufan.
What next?
Execute steps 1-3. If everything goes ok. Execute step 4 for gitlab-ce/www-gitlab-com, etc. If everything goes OK execute step 5.