[meta] Current state of Object Storage on GitLab.com
This issue presents current configuration and migration status of file storage to object storage on GitLab.com
Production
| Type | Object Storage | Status |
|---|---|---|
| Artifacts | Configured and background upload enabled (S3) | All objects on Object Storage, and automatic migration enabled |
| Traces | Configured and background upload enabled (S3) | All traces are on Object Storage |
| LFS | Configured and direct_upload enabled (GCS) | All LFS objects are migrated |
| Uploads | Configured (GCS) | Some uploads migrated (specifically gitlab-com and gitlab-org groups) |
Canary
| Type | Object Storage | Status |
|---|---|---|
| Artifacts | Configured and background upload enabled (S3) | All objects on Object Storage, and automatic migration enabled |
| Traces | Configured and background upload enabled (S3) | All traces are on Object Storage. |
| LFS | Configured and direct_upload (GCS) |
All LFS objects on object storage |
| Uploads | Configured (GCS) | Some uploads migrated (specifically gitlab-com and gitlab-org groups) |
Staging
| Type | Object Storage | Status |
|---|---|---|
| Artifacts | Configured and background upload enabled (S3) | All objects on Object Storage, and automatic migration enabled |
| Traces | Configured and background upload enabled (S3) | All recent traces (after 2018-02-10) are on Object Storage (Traces are stored as Artifacts) |
| LFS | Configured and direct_upload enabled (GCS) | All objects migrated |
| Uploads | Configured and background_upload enabled (GCS) | All objects migrated |
Timeline
2018-02-20
- Production status:
- We queued all OS of artifacts, All new are on OS,
- LFS, done nothing: 10 objects migrated,
- Uploads, done nothing: we have bucket, we don't have application configured,
- Traces:
- Old traces: 50.000.000 -> to migrate,
- Live traces: traces of running jobs -> nothing to do,
- New traces: for completed jobs -> on OS,
- Development status:
- We are working on migration resiliency: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4518, https://gitlab.com/gitlab-org/gitlab-ee/issues/4980
- Uploads are currently blocked by https://gitlab.com/gitlab-org/gitlab-ee/issues/4952
2018-02-28
- Canary status:
- enabled artifacts and LFS object storage on canary. This should have been done when production was changed, but it was overlooked.
2018-03-16
- Migrate 100,000
gitlab-cetraces to object storage
2018-03-20
- Enable LFS object storage with JSON API on staging.
Edited by Alex Hanselka