Categorize existing Object Storage buckets
As part of the Object Storage Working Group we want to document the current state of the Object Storage in GitLab.
- Identify Bucket Ownership - internal google doc
- Categorize how features interact with the Object Storage
Categorization
We thought it was possible to categorize our Object Storage interaction following the three upload technologies defined in our [documentation].
After digging into this it was clear that those categories do not cover the entire spectrum of features we have.
| Feature | upload technology | uploader | bucket structure |
|---|---|---|---|
| job artifacts | direct_upload | workhorse | /artifacts/<proj_id_hash>/<date>/<job_id>/<artifact_id> |
| pipeline artifacts | carrierwave | sidekiq | /artifacts/<proj_id_hash>/pipelines/<pipeline_id>/artifacts/<artifact_id> |
| live job traces | fog | sidekiq | /artifacts/tmp/builds/<job_id>/chunks/<chunk_index>.log |
| job traces archive | carrierwave | sidekiq | /artifacts/<proj_id_hash>/<date>/<job_id>/<artifact_id>/job.log |
| autoscale runner caching | N/A | gitlab-runner | /gitlab-com-[platform-]runners-cache/??? |
| backups | N/A | s3cmd, awscli, or gcs | /gitlab-backups/??? |
| git LFS | direct_upload | workhorse | /lsf-objects/<lfs_obj_oid[0:2]>/<lfs_obj_oid[2:2]> |
| design management files | disk buffering | rails controller | /lsf-objects/<lfs_obj_oid[0:2]>/<lfs_obj_oid[2:2]> |
| design management thumbnails | carrierwave | sidekiq | /uploads/design_management/action/image_v432x230/<model_id> |
| generic file uploads | direct_upload | workhorse | /uploads/@hashed/[0:2]/[2:4]/<hash1>/<hash2>/file |
| generic file uploads - personal snippets | direct_upload | workhorse | /uploads/personal_snippet/<snippet_id>/<filename> |
| global appearance settings | disk buffering | rails controller | /uploads/appearance/... |
| topics | disk buffering | rails controller | /uploads/projects/topic/... |
| Avatar images | direct_upload | workhorse | /uploads/[user,group,project]/avatar/<model_id> |
| Import/export | direct_upload | workhorse | /uploads/import_export_upload/??? |
| GitLab Migration | carrierwave | sidekiq | /uploads/bulk_imports/??? |
| MR diffs | carrierwave (?) | sidekiq (?) | /external-diffs/merge_request_diffs/mr-<mr_id>/diff-<diff_id> |
| package manager archives | direct_upload | sidekiq | /packages/<proj_id_hash>/packages/<pkg_segment>/files/<pkg_file_id> |
| package manager archives | direct_upload | sidekiq | /packages/<container_id_hash>/debian_*_component_file/<component_file_id> |
| package manager archives | direct_upload | sidekiq | /packages/<container_id_hash>/debian_*_distribution/<distribution_file_id> |
| container image cache (?) | direct_upload | workhorse | /dependency-proxy/<group_id_hash>/dependency_proxy/<group_id>/files/<proxy_id>/<blob_id or manifest_id> |
| terraform state files | carrierwave | rails controller | /terraform/<proj_id_hash>/<terraform_state_id> |
| Pages content archives | carrierwave | sidekiq | /gitlab-gprd-pages/<proj_id_hash>/pages_deployments/<deployment_id>/ |
NOTE: do not edit this table directly in the issue, edit the internal google sheet and cut and paste here.
CarrierWave interactions
The following table contains the list of files interacting with carrierwave, it is a good indicator of the number of features as well as a checkpoint for the categorization table above.
| File | Carrierwave usage | Categorized |
|---|---|---|
app/models/project.rb |
include Avatarable |
|
app/models/projects/topic.rb |
include Avatarable |
|
app/models/group.rb |
include Avatarable |
|
app/models/user.rb |
include Avatarable |
|
app/models/terraform/state_version.rb |
include FileStoreMounter |
|
app/models/ci/job_artifact.rb |
include FileStoreMounter |
|
app/models/ci/pipeline_artifact.rb |
include FileStoreMounter |
|
app/models/pages_deployment.rb |
include FileStoreMounter |
|
app/models/lfs_object.rb |
include FileStoreMounter |
|
app/models/dependency_proxy/blob.rb |
include FileStoreMounter |
|
app/models/dependency_proxy/manifest.rb |
include FileStoreMounter |
|
app/models/packages/composer/cache_file.rb |
include FileStoreMounter |
|
app/models/packages/package_file.rb |
include FileStoreMounter |
|
app/models/concerns/packages/debian/component_file.rb |
include FileStoreMounter |
|
ee/app/models/issuable_metric_image.rb |
include FileStoreMounter |
|
ee/app/models/vulnerabilities/remediation.rb |
include FileStoreMounter |
|
ee/app/models/vulnerabilities/export.rb |
include FileStoreMounter |
|
app/models/packages/debian/project_distribution.rb |
include Packages::Debian::Distribution |
|
app/models/packages/debian/group_distribution.rb |
include Packages::Debian::Distribution |
|
app/models/packages/debian/project_component_file.rb |
include Packages::Debian::ComponentFile |
|
app/models/packages/debian/group_component_file.rb |
include Packages::Debian::ComponentFile |
|
app/models/merge_request_diff.rb |
mount_uploader :external_diff, ExternalDiffUploader |
|
app/models/note.rb |
mount_uploader :attachment, AttachmentUploader |
|
app/models/appearance.rb |
mount_uploader :logo, AttachmentUploader |
|
app/models/appearance.rb |
mount_uploader :header_logo, AttachmentUploader |
|
app/models/appearance.rb |
mount_uploader :favicon, FaviconUploader |
|
app/models/project.rb |
mount_uploader :bfg_object_map, AttachmentUploader |
|
app/models/import_export_upload.rb |
mount_uploader :import_file, ImportExportUploader |
|
app/models/import_export_upload.rb |
mount_uploader :export_file, ImportExportUploader |
|
app/models/ci/deleted_object.rb |
mount_uploader :file, DeletedObjectUploader |
|
app/models/design_management/action.rb |
mount_uploader :image_v432x230, DesignManagement::DesignV432x230Uploader |
|
app/models/concerns/packages/debian/distribution.rb |
mount_uploader :signed_file, Packages::Debian::DistributionReleaseFileUploader |
|
app/models/bulk_imports/export_upload.rb |
mount_uploader :export_file, ExportUploader |
|
ee/app/models/user_permission_export_upload.rb |
mount_uploader :file, AttachmentUploader |
NOTE: do not edit this table directly in the issue, edit the internal google sheet and cut and paste here.
Edited by Matthias Käppler