Investigate why artifacts are going to disk and migrate them to object storage
During the course of investigation into https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6467, the share-01 server filling up, we discovered that there are a substantial amount of artifacts on disk.
@dosuken123 provided the following numbers:
gitlabhq_production=> SELECT count(*) from ci_job_artifacts where file_store = 1;
count
--------
819729
(1 row)
gitlabhq_production=> SELECT file_type, count(*) From ci_job_artifacts Where file_store = 1 AND id < 140000000 AND id > 130000000 GROUP BY file_type;
file_type | count
-----------+--------
2 | 303602
1 | 303565
(2 rows)
I just checked again to be sure and the number does not appear to be growing. In fact it fell by 10 since @dosuken123 gathered his numbers on 2019-03-28.
gitlabhq_production=> SELECT count(*) from ci_job_artifacts where file_store = 1;
count
--------
819719
(1 row)
We need to determine how so many artifacts got left on disk, ensure that this doesn't happen again, and migrate the artifacts on disk to object storage.
- previous migration issue: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/2387