Statistics artifacts_size is broken (#5380) · Epics · GitLab.org

Statistics artifacts_size is broken

Recent changes in statistics have broken artifacts_size counters. We have started seeing really huge and also negative numbers on our local instance. Example: ``` "statistics":{ "commit_count":256, "storage_size":11204154550, "repository_size":18800967, "lfs_objects_size":0, "job_artifacts_size":-115028208 } ``` Following `gitlab-rails` script scanned our instance and has found 1097 non-matching counts and 28 negative numbers and that out of ~5.5k projects with builds, which is ~20% with wrong statistics. If we would count only those building really often, the rate is much higher. ``` ::Project.with_statistics().find_each do |project| size_real = Ci::JobArtifact.artifacts_size_for(project).to_i size_old = project.builds.sum(:artifacts_size).to_i # should be 0 after migrations size_now = project.statistics.build_artifacts_size puts "Project #{project.id}: #{project.path_with_namespace} -> real = #{size_real}; old = #{size_old}; now = #{size_now}" puts " ERROR - OLD SIZE" if size_old != 0 puts " ERROR - SIZE MISMATCH" if size_real + size_old != size_now puts " ERROR - NEGATIVE VALUE" if size_now < 0 end ``` During analysis of recent gitlab-ce code changes, I was only able to identify 1 problem * [x] updating `artifacts_size` does not trigger `storage_size` update, so `storage_size` will be lower than sum of components if CI is triggered; -> MR is available: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20697 One noticeable thing it that no project created after migration 10.7.3 -> 10.8.2 shows the problem (yet?). So it is possible that the code is good now, just stats are broken due to legacy stuff or bugs in previous releases and they are not able to self-heal because the new code just adds/removes bytes and never recalculates the value. * [x] ~~fix wrong counting of `job_artifacts_size`~~ -> assuming this is legacy problem * [x] **implement migration script to recalculate artifact sizes** Investigations took longer than I'd like to also due to following issue: * [x] jobs API provides only `artifacts_file.size`, however job_artifacts_size counts all artifact types (artifact, trace, metadata) -> MR is available: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20821 * [x] GUI does not show (unless you open each single job) if the trace is available and does not provide any deletion possibility -> MR is available: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/18707 So the user is not able to see which job is consuming the space and even admin has problems finding that out. With 100k jobs in a single project, I have to do hell of a scripting to clean up while being blind at the same time (not knowing if the job being processed actually has trace or not). And finally the reason why we started the investigations: After migration to 11.0.1 storage consumption statistics of some projects have sky-rocketed from <10G to >30G. We were able to identify that this was due to legacy trace archiving (manual, since automatic migrations have failed, see gitlab-com/infrastructure#4377). Suddenly the traces count towards storage consumption stats and with >100k traces within a project it's huge increase. It good to count these, however it just added too much in a single shot. Most of our projects have the traces much larger than artifacts: - people love quick analysis of failed builds so maximum logging is usually enabled - artifacts self-delete themselves after configured time - artifacts (usually binaries) are compressed while traces (clear text, the best format for compression) not We think that following two points make the life easier for both admin and user: * [x] ~~set expiration of traces~~ -> trace management as mentioned above is enough * [x] ~~compress traces the same way as artifacts~~ -> extracted to https://gitlab.com/gitlab-org/gitlab/-/issues/23678 And finally - why are there still legacy artifacts and metadata? Traces were migrated in 11.0, shouldn't the other types be migrated now, too? These could contribute to the errors in counting... * [x] ~~migrate legacy artifacts and metadata~~ -> handled under https://gitlab.com/gitlab-org/gitlab-foss/-/issues/46652 / https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/18615 Relevant MRs I have found for recent changes: - https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/14367 - add NEW traces and metadata to statistics, but do NOT show these anywhere - https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/16539 - bugfix for artifacts_size calculation (increment/decrement cannot fix these) - https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17839 - artifacts size is not fully refreshed anymore, just incremented/decremented Feel free to split this issue to smaller pieces reporting/fixing each problem/proposal separately. ## Proposal 1. ~~Make `build_artifacts_size` use [efficient counters](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35878).~~ 1. Enable `efficient_counter_attribute` feature flag and monitor it https://gitlab.com/gitlab-org/gitlab/-/issues/238535 1. Fix decrement of `build_artifact_size` when artifacts are deleted: https://gitlab.com/gitlab-org/gitlab/-/issues/224151 1. Write a script that can forcefully recalculate all artifacts size (e.g. a rake task that iterates over projects) and document how to use it. https://gitlab.com/gitlab-org/gitlab/-/issues/238536 1. Run a database migration to update values using the script https://gitlab.com/gitlab-org/gitlab/-/issues/332994 1. Monitor for long running transactions and pickup https://gitlab.com/gitlab-org/gitlab/-/issues/29070 if they are still present after work above is done.  *This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.*

epic