Expired artifacts still taking disk space
## **If you are experiencing this bug, upgrade to 14.9**
This issue will be closed when we implement back ports for 14.6,14.7, and 14.8
### Summary
We are experiencing high disk usage in the artifact storage (local) even if artifacts are correctly marked as expired. In the UI we see that some build artifacts are no longer available, but they are still present on disk.
### Steps to reproduce
I have no easy reproduction step as the issue seems to be happening only on some builds, apparently by random chance. I can run any additional check you may suggest in order to better understand what's happening.
As a side note, we updated to Gitlab 14.4 just today. Those older builds were executed on Gitlab 13.10.
### What is the current *bug* behavior?
Most of my jobs have the `expire_in: 2h` clause - the rest are marked with `expire_in: 1 week` . No artifact should ever be stored for more than a week. Build artifacts seem to expire correctly, but sometimes they are not removed from disk.
### What is the expected *correct* behavior?
Artifacts should be removed shortly after expiration, reclaiming their used disk space.
### Relevant logs and/or screenshots
The `ExpireBuildArtifactsWorker` cronjob is running correctly every 7 minutes (the default scheduling):
```
# cat /var/log/gitlab/sidekiq/current | grep ExpireBuildArtifactsWorker
{"severity":"INFO","time":"2021-11-22T12:00:07.834Z","queue":"cronjob:expire_build_artifacts","args":[],"class":"ExpireBuildArtifactsWorker","retry":0,"version":0,"queue_namespace":"cronjob","jid":"1a39b76097110c3ef7badcde","created_at":"2021-11-22T12:00:07.720Z","meta.caller_id":"Cronjob","correlation_id":"d587da0be9acf30a9d8317b13da780df","idempotency_key":"resque:gitlab:duplicate:cronjob:expire_build_artifacts:7e864f760219ae33143ba929fea73d2c9117919da80b189f87db84994980be48","enqueued_at":"2021-11-22T12:00:07.721Z","job_size_bytes":2,"pid":1644679,"message":"ExpireBuildArtifactsWorker JID-1a39b76097110c3ef7badcde: start","job_status":"start","scheduling_latency_s":0.113121}
{"severity":"INFO","time":"2021-11-22T12:00:07.951Z","class":"Namespaces::ScheduleAggregationWorker","args":["113"],"retry":3,"queue":"update_namespace_statistics:namespaces_schedule_aggregation","version":0,"queue_namespace":"update_namespace_statistics","jid":"de3e0a811e407584784417ca","created_at":"2021-11-22T12:00:07.948Z","meta.caller_id":"ExpireBuildArtifactsWorker","meta.feature_category":"continuous_integration","meta.client_id":"ip/","correlation_id":"d587da0be9acf30a9d8317b13da780df","idempotency_key":"resque:gitlab:duplicate:update_namespace_statistics:namespaces_schedule_aggregation:a48ed0441003298938285446a1beea7d3de501a28f4d771fef87bc1ae9c7ddc1","enqueued_at":"2021-11-22T12:00:07.949Z","job_size_bytes":5,"pid":1644679,"message":"Namespaces::ScheduleAggregationWorker JID-de3e0a811e407584784417ca: start","job_status":"start","scheduling_latency_s":0.001676}
{"severity":"INFO","time":"2021-11-22T12:00:07.961Z","class":"Namespaces::ScheduleAggregationWorker","args":["113"],"retry":3,"queue":"update_namespace_statistics:namespaces_schedule_aggregation","version":0,"queue_namespace":"update_namespace_statistics","jid":"de3e0a811e407584784417ca","created_at":"2021-11-22T12:00:07.948Z","meta.caller_id":"ExpireBuildArtifactsWorker","meta.feature_category":"continuous_integration","meta.client_id":"ip/","correlation_id":"d587da0be9acf30a9d8317b13da780df","idempotency_key":"resque:gitlab:duplicate:update_namespace_statistics:namespaces_schedule_aggregation:a48ed0441003298938285446a1beea7d3de501a28f4d771fef87bc1ae9c7ddc1","enqueued_at":"2021-11-22T12:00:07.949Z","job_size_bytes":5,"pid":1644679,"message":"Namespaces::ScheduleAggregationWorker JID-de3e0a811e407584784417ca: done: 0.010041 sec","job_status":"done","scheduling_latency_s":0.001676,"redis_calls":2,"redis_duration_s":0.000381,"redis_read_bytes":2,"redis_write_bytes":228,"redis_queues_calls":2,"redis_queues_duration_s":0.000381,"redis_queues_read_bytes":2,"redis_queues_write_bytes":228,"db_count":2,"db_write_count":0,"db_cached_count":0,"cpu_s":0.004016,"mem_objects":998,"mem_bytes":34056,"mem_mallocs":141,"mem_total_bytes":73976,"duration_s":0.010041,"completed_at":"2021-11-22T12:00:07.961Z","db_duration_s":0.004639}
{"severity":"INFO","time":"2021-11-22T12:00:09.807Z","queue":"cronjob:expire_build_artifacts","args":[],"class":"ExpireBuildArtifactsWorker","retry":0,"version":0,"queue_namespace":"cronjob","jid":"1a39b76097110c3ef7badcde","created_at":"2021-11-22T12:00:07.720Z","meta.caller_id":"Cronjob","correlation_id":"d587da0be9acf30a9d8317b13da780df","idempotency_key":"resque:gitlab:duplicate:cronjob:expire_build_artifacts:7e864f760219ae33143ba929fea73d2c9117919da80b189f87db84994980be48","enqueued_at":"2021-11-22T12:00:07.721Z","job_size_bytes":2,"pid":1644679,"message":"ExpireBuildArtifactsWorker JID-1a39b76097110c3ef7badcde: done: 1.972892 sec","job_status":"done","scheduling_latency_s":0.113121,"redis_calls":6,"redis_duration_s":0.006063000000000001,"redis_read_bytes":265,"redis_write_bytes":1822,"redis_cache_calls":1,"redis_cache_duration_s":0.000126,"redis_cache_read_bytes":203,"redis_cache_write_bytes":62,"redis_queues_calls":3,"redis_queues_duration_s":0.005629,"redis_queues_read_bytes":60,"redis_queues_write_bytes":1456,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.000308,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":304,"db_count":2009,"db_write_count":5,"db_cached_count":1988,"cpu_s":1.794631,"mem_objects":1135121,"mem_bytes":17260672,"mem_mallocs":254581,"mem_total_bytes":62665512,"extra.expire_build_artifacts_worker.destroyed_job_artifacts_count":2,"duration_s":1.972892,"completed_at":"2021-11-22T12:00:09.807Z","db_duration_s":0.131203}
```
### Output of checks
The orphan cleanup job does not find anything relevant:
```
lscorcia@MOPAPGITLAB01:~$ sudo gitlab-rake gitlab:cleanup:orphan_job_artifact_files DRY_RUN=false
I, [2021-11-22T12:48:51.036823 #1676957] INFO -- : Looking for orphan job artifacts to clean up
I, [2021-11-22T12:48:51.037377 #1676957] INFO -- : find command: '/bin/ionice -c best-effort find -L /var/opt/gitlab/gitlab-rails/shared/artifacts -mindepth 6 -maxdepth 6 -type d'
I, [2021-11-22T12:48:51.486441 #1676957] INFO -- : Processed 12955 job artifact(s) to find and cleaned 0 orphan(s).
```
However, if I run the following script in the Rails console, I see a bunch of items that should be expired:
```
project = Project.find_by_full_path('mypath/myproject')
builds_with_artifacts = project.builds.with_downloadable_artifacts
builds_to_clear = builds_with_artifacts.where("finished_at < ?", 2.week.ago)
builds_to_clear.find_each do |build|
puts build.artifacts_expire_at
end
```
```
2021-08-25 18:37:31 UTC
2021-08-25 18:35:05 UTC
2021-09-01 16:47:07 UTC
2021-08-25 20:41:45 UTC
2021-08-25 20:43:59 UTC
2021-09-01 18:54:07 UTC
2021-08-26 21:53:23 UTC
2021-08-26 21:51:33 UTC
2021-09-02 20:03:35 UTC
2021-09-01 15:57:52 UTC
2021-09-01 16:00:44 UTC
2021-09-08 14:11:36 UTC
2021-09-01 17:48:03 UTC
2021-09-01 17:50:27 UTC
2021-09-08 16:01:06 UTC
2021-09-02 14:28:22 UTC
2021-09-02 14:50:46 UTC
2021-09-09 13:00:37 UTC
2021-09-03 15:43:30 UTC
2021-09-03 15:45:52 UTC
2021-09-10 13:56:03 UTC
2021-09-08 20:26:32 UTC
2021-09-08 20:29:04 UTC
2021-09-15 18:38:52 UTC
2021-10-19 23:31:51 UTC
2021-10-19 23:34:44 UTC
2021-10-20 20:35:31 UTC
2021-10-20 20:42:10 UTC
2021-10-20 20:43:55 UTC
2021-10-26 23:32:33 UTC
2021-10-26 23:35:55 UTC
2021-10-26 23:27:55 UTC
```
However, in the UI those very same builds are marked as `No artifacts found` (see attached image) in the Pipelines screen:
This is a report of disk usage for the `/var/opt/gitlab/gitlab-rails/shared/artifacts` folder:
```
# du -a . | sort -n -r | head -n 50
74653360 .
28496036 ./86
28496032 ./86/e5
28496028 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef
25633164 ./8e
25633160 ./8e/61
25633156 ./8e/61/8e612bd1f5d132a339575b8dafb7842c64614e56bcf3d5ab65a0bc4b34329407
5489668 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef/2021_08_25
5485552 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef/2021_09_01
4207000 ./8c
4206996 ./8c/d2
4206992 ./8c/d2/8cd2510271575d8430c05368315a87b9c4784c7389a47496080c1e615a2a00b6
2762496 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef/2021_09_02
2758380 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef/2021_09_08
2754876 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef/2021_09_03
2743088 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef/2021_08_26
1984740 ./8e/61/8e612bd1f5d132a339575b8dafb7842c64614e56bcf3d5ab65a0bc4b34329407/2021_11_12
1751148 ./4a
1751144 ./4a/85
1751140 ./4a/85/4a8596a7790b5ca9e067da401c018b3206befbcf95c38121854d1a0158e7678a
1699104 ./73
1690196 ./73/4d
1690192 ./73/4d/734d0759cdb4e0d0a35e4fd73749aee287e4fdcc8648b71a8d6ed591b7d4cb3f
1651024 ./cb
1651020 ./cb/a2
1651016 ./cb/a2/cba28b89eb859497f544956d64cf2ecf29b76fe2ef7175b33ea59e64293a4461
1612408 ./5e
1612404 ./5e/f6
1612400 ./5e/f6/5ef6fdf32513aa7cd11f72beccf132b9224d33f271471fff402742887a171edf
1586532 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef/2021_11_22
1512348 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef/2021_10_26
1501992 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef/2021_10_20
1479536 ./65
1479532 ./65/a6
1479528 ./65/a6/65a699905c02619370bcf9207f5a477c3d67130ca71ec6f750e07fe8d510b084
1400136 ./7b
1400132 ./7b/69
1400128 ./7b/69/7b69759630f869f2723875f873935fed29d2d12b10ef763c1c33b8e0004cb405
1367336 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef/2021_09_08/28367
1367144 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef/2021_09_08/28367/43117
```
#### Results of GitLab environment info
GitLab Community on-premises setup.
<details>
<summary>Expand for output related to GitLab environment info</summary>
<pre>
System information
System: Ubuntu 20.04
Current User: git
Using RVM: no
Ruby Version: 2.7.4p191
Gem Version: 3.1.4
Bundler Version:2.1.4
Rake Version: 13.0.6
Redis Version: 6.0.16
Git Version: 2.33.0.
Sidekiq Version:6.2.2
Go Version: unknown
GitLab information
Version: 14.4.2
Revision: 1ce86e92f81
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 12.7
URL: https://gitlab.mydomain.com
HTTP Clone URL: https://gitlab.mydomain.com/some-group/some-project.git
SSH Clone URL: git@gitlab.mydomain.com:some-group/some-project.git
Using LDAP: yes
Using Omniauth: yes
Omniauth Providers: openid_connect
GitLab Shell
Version: 13.21.1
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/git
</pre>
</details>
#### Results of GitLab application Check
<details>
<summary>Expand for output related to the GitLab application check</summary>
<pre>
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Checking Reply by email ...
IMAP server credentials are correct? ... Checking gitlab@mydomain.com
yes
Init.d configured correctly? ... skipped
MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 100 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
3/6 ... yes
3/7 ... yes
3/15 ... yes
3/16 ... yes
3/24 ... yes
3/25 ... yes
3/32 ... yes
36/33 ... yes
3/34 ... yes
8/35 ... yes
3/36 ... yes
3/37 ... yes
3/38 ... yes
3/39 ... yes
3/41 ... yes
70/42 ... yes
3/43 ... yes
16/44 ... yes
18/45 ... yes
18/46 ... yes
18/47 ... yes
23/49 ... yes
23/50 ... yes
23/51 ... yes
3/56 ... yes
34/57 ... yes
3/60 ... yes
42/61 ... yes
45/62 ... yes
51/65 ... yes
51/66 ... yes
51/67 ... yes
51/68 ... yes
51/69 ... yes
55/71 ... yes
119/72 ... yes
55/73 ... yes
55/74 ... yes
55/75 ... yes
55/76 ... yes
55/77 ... yes
55/78 ... yes
3/79 ... yes
3/80 ... yes
3/81 ... yes
3/82 ... yes
3/83 ... yes
59/87 ... yes
55/88 ... yes
53/89 ... yes
28/91 ... yes
51/92 ... yes
51/94 ... yes
119/95 ... yes
3/96 ... yes
3/97 ... yes
51/98 ... yes
77/100 ... yes
3/101 ... yes
3/104 ... yes
3/105 ... yes
3/106 ... yes
3/107 ... yes
3/109 ... yes
3/110 ... yes
3/111 ... yes
77/112 ... yes
3/113 ... yes
119/114 ... yes
119/115 ... yes
3/116 ... yes
3/117 ... yes
3/118 ... yes
47/119 ... yes
3/120 ... yes
3/121 ... yes
36/122 ... yes
84/123 ... yes
3/126 ... yes
3/127 ... yes
3/128 ... yes
3/129 ... yes
3/130 ... yes
3/131 ... yes
3/132 ... yes
3/133 ... yes
3/136 ... yes
3/137 ... yes
3/138 ... yes
3/139 ... yes
88/140 ... yes
88/142 ... yes
3/143 ... yes
93/145 ... yes
93/148 ... yes
3/149 ... yes
3/150 ... yes
88/151 ... yes
3/152 ... yes
102/153 ... yes
102/157 ... yes
102/158 ... yes
51/159 ... yes
102/160 ... yes
3/161 ... yes
3/162 ... yes
77/164 ... yes
3/165 ... yes
3/166 ... yes
3/167 ... yes
113/168 ... yes
3/170 ... yes
3/171 ... yes
50/172 ... yes
3/173 ... yes
119/174 ... yes
3/175 ... yes
102/176 ... yes
23/177 ... yes
3/178 ... yes
65/179 ... yes
102/180 ... yes
3/181 ... yes
102/182 ... yes
119/183 ... yes
119/184 ... yes
3/185 ... yes
131/186 ... yes
102/187 ... yes
3/188 ... yes
3/189 ... yes
3/190 ... yes
88/191 ... yes
3/192 ... yes
3/193 ... yes
47/194 ... yes
143/196 ... yes
143/197 ... yes
143/198 ... yes
3/199 ... yes
51/200 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.4)
Git version >= 2.33.0 ? ... yes (2.33.0)
Git user has default SSH configuration? ... yes
Active users: ... 112
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
</pre>
</details>
### Possible fixes
<!-- If you can, link to the line of code that might be responsible for the problem. -->
issue