Background migration to fix incorrect statistics for Packages storage
Context
In #363010 (closed), we uncovered an issue with the Package storage statistics calculation. #363010 (closed) fixed the issue moving forward by changing how the packages_size
metric is handled and using a counter attribute
. This allows the updates to be "stacked" in Redis and flushed after an interval to the database = reduction in chances to have a concurrent update on the project statistics object.
Problem to solve
The problem is that the above fix only fixed the issue moving forward. We still have 100k projects that used the old, incorrect method. This data needs to be fixed prior to enforcing storage limits.
Proposal
Fix the existing data, so that storage calculations for the Package Registry are correct for all GitLab customers.
Further details
We might need more time to evaluate if we can use a (background) migration.