how to recalculate project statistics build artifacts size?
Summary
Customer contacted Support about their 15.5 instance as, having cleared down pipelines and artifacts, one project still showed a large number for build artifact size. GitLab team members can read more in the ticket.
irb(main):001:0> Project.find_by_id(87277).statistics.build_artifacts_size
=> 32307570098
But checking the database
select project_id, file_type, locked, count(*), sum(size) as size
from ci_job_artifacts
where project_id in (87277)
group by project_id,file_type, locked having count(*) > 1
order by size desc;
It should be 7gb-8gb
project_id | redacted | locked | redacted | size
------------+-----------+--------+----------+------------
87277 | | 1 | | 7315871627
87277 | | 1 | | 21177632
87277 | | 2 | | 4394915
87277 | | 0 | | 4228439
87277 | | 1 | | 213597
87277 | | 1 | | 3269
How can this be recalculated?
-
In !41769 (merged)
build_artifacts_size
was changed to acounter_attribute
and looking at the project statistics model it seems likeProject.find_by_id(xx).statistics.refresh!
won't touchbuild_artifacts_size
since it's not inCOLUMNS_TO_REFRESH
-
#238536 (closed) is closed in %14.10, having specified:
write a script that can forcefully recalculate all artifacts size and document how to use it.
- Where is this documented?
-
I see the merge request !81306 (merged) specified
rake gitlab:project_with_incorrect_artifact_size | rake gitlab:update_project_statistics_build_artifacts_size
-
However the first part of that chain doesn't exist in %15.4
gitlab-rake gitlab:project_with_incorrect_artifact_size rake aborted! Don't know how to build task 'gitlab:project_with_incorrect_artifact_size' (See the list of available tasks with `rake --tasks`) Did you mean? gitlab:refresh_project_statistics_build_artifacts_size
-
gitlab-rake -T
now shows that a HTTP URL needs to be passed togitlab:refresh_project_statistics_build_artifacts_size
gitlab:refresh_project_statistics_build_artifacts_size[csv_url]
-
Having found a way to reset a project's artifacts size to zero ..
Project.find_by_id(87).statistics.reset_counter!(:build_artifacts_size)
-
I found I could get around this requirement to serve up a CSV on a website with:
projects = Project.where(id: 87) Projects::BuildArtifactsSizeRefresh.enqueue_refresh(projects)
-
How can we then complete this work? I've tried doing this:
# having run `enqueue_refresh` # shows a state: 1 (created) pp Projects::BuildArtifactsSizeRefresh.remaining Projects::RefreshBuildArtifactsSizeStatisticsWorker.new.perform_work # shows a state: 3 (pending) pp Projects::BuildArtifactsSizeRefresh.remaining
I'm finding that if I wait a bit, the statistics get updated. On my test instance. Is there a way we can complete this work synchronously?
Steps to reproduce
As above. Project statistics do not match ci_job_artifacts
Example Project
What is the current bug behavior?
Self-managed customer seeing issues where artifact size is wrong. Cannot find a documented way to fix this.
What is the expected correct behavior?
Relevant logs and/or screenshots
Output of checks
15.4