Script that recalculates project's total build artifacts size does not factor in effects of artifact deletion
Summary
After attempting to run the rake task from #238536 (closed) on .com twice (ref gitlab-com/gl-infra/production#6631 (comment 909726250)), we still have a lot of projects with inaccurate stats based on https://app.periscopedata.com/app/gitlab/983187/CI-Build-Artifact-Size-Analysis.
@alberts-gitlab brought up one bug that could play a big factor on the inaccurate stats:
We don't completely factor in the effects of when artifacts are deleted:
During the refresh, if one artifact is deleted, it will decrement the counter. But the baseline accumulation will not include that artifact.
So say we have 11 artifacts, 1Mb each, and batch size of 10.
We start with counter at 0 after clearing it. As we sum the first 10, the 11th artifact is deleted. This means we have 10Mb accumulated, with -1 Mb in the counter. The Sum ends at the 10th artifact, because there is no more 11th artifact.
Then the counter is flushed, leaving 9Mb in total instead of the expected 10Mb.
With the ongoing #356224 (closed), this bug may have played a huge factor with the inaccurate recalculated stats.
Possible fixes
TBD