[meta] Current state of Object Storage on GitLab.com
This issue presents current configuration and migration status of file storage to object storage on GitLab.com
Production
Type |
Object Storage |
Status |
Artifacts |
Configured and background upload enabled (S3) |
All objects on Object Storage, and automatic migration enabled |
Traces |
Configured and background upload enabled (S3) |
All traces are on Object Storage |
LFS |
Configured and direct_upload enabled (GCS) |
All LFS objects are migrated |
Uploads |
Configured (GCS) |
Some uploads migrated (specifically gitlab-com and gitlab-org groups) |
Canary
Type |
Object Storage |
Status |
Artifacts |
Configured and background upload enabled (S3) |
All objects on Object Storage, and automatic migration enabled |
Traces |
Configured and background upload enabled (S3) |
All traces are on Object Storage. |
LFS |
Configured and direct_upload (GCS) |
All LFS objects on object storage |
Uploads |
Configured (GCS) |
Some uploads migrated (specifically gitlab-com and gitlab-org groups) |
Staging
Type |
Object Storage |
Status |
Artifacts |
Configured and background upload enabled (S3) |
All objects on Object Storage, and automatic migration enabled |
Traces |
Configured and background upload enabled (S3) |
All recent traces (after 2018-02-10) are on Object Storage (Traces are stored as Artifacts) |
LFS |
Configured and direct_upload enabled (GCS) |
All objects migrated |
Uploads |
Configured and background_upload enabled (GCS) |
All objects migrated |
Timeline
2018-02-20
- Production status:
- We queued all OS of artifacts, All new are on OS,
- LFS, done nothing: 10 objects migrated,
- Uploads, done nothing: we have bucket, we don't have application configured,
- Traces:
- Old traces: 50.000.000 -> to migrate,
- Live traces: traces of running jobs -> nothing to do,
- New traces: for completed jobs -> on OS,
- Development status:
2018-02-28
- Canary status:
- enabled artifacts and LFS object storage on canary. This should have been done when production was changed, but it was overlooked.
2018-03-16
- Migrate 100,000
gitlab-ce
traces to object storage
2018-03-20
- Enable LFS object storage with JSON API on staging.
Edited
by
Alex Hanselka