Skip to content

Backfill project stats to remove pipeline artifact

Vijay Hawoldar requested to merge vij-backfill-ps-pipeline-artifacts into master

What does this MR do and why?

As described in https://gitlab.com/gitlab-org/gitlab/-/issues/412634, we need to ensure total storage sizes for ProjectStatistics and Namespaces::RootStorageStatistics are updated to remove pipeline_artifacts_size.

In this MR we achieve that by:

  1. looking up ProjectStatistics that have a pipeline_artifacts_size > 0
  2. re-calculating the storage_size based on the component parts that make up the value, but without pipeline_artifacts_size
  3. trigger the async worker that handles the RootStorageStatistics update that will exclude the same (that class sums the related project_statistic#storage_size values, so being excluded from one means it'll be excluded from the other)

We need to have an EE override for this background migration to handle a nuance with ProjectStatistics that means we also need to exclude upload_size when calculating the new total.

This MR has used Backfill to exclude uploads_size from storage_s... (!103017 - merged) as the basis for the implementation

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Vijay Hawoldar

Merge request reports