Expired artifacts still taking disk space
If you are experiencing this bug, upgrade to 14.9
This issue will be closed when we implement back ports for 14.6,14.7, and 14.8
Summary
We are experiencing high disk usage in the artifact storage (local) even if artifacts are correctly marked as expired. In the UI we see that some build artifacts are no longer available, but they are still present on disk.
Steps to reproduce
I have no easy reproduction step as the issue seems to be happening only on some builds, apparently by random chance. I can run any additional check you may suggest in order to better understand what's happening.
As a side note, we updated to Gitlab 14.4 just today. Those older builds were executed on Gitlab 13.10.
What is the current bug behavior?
Most of my jobs have the expire_in: 2h
clause - the rest are marked with expire_in: 1 week
. No artifact should ever be stored for more than a week. Build artifacts seem to expire correctly, but sometimes they are not removed from disk.
What is the expected correct behavior?
Artifacts should be removed shortly after expiration, reclaiming their used disk space.
Relevant logs and/or screenshots
The ExpireBuildArtifactsWorker
cronjob is running correctly every 7 minutes (the default scheduling):
# cat /var/log/gitlab/sidekiq/current | grep ExpireBuildArtifactsWorker
{"severity":"INFO","time":"2021-11-22T12:00:07.834Z","queue":"cronjob:expire_build_artifacts","args":[],"class":"ExpireBuildArtifactsWorker","retry":0,"version":0,"queue_namespace":"cronjob","jid":"1a39b76097110c3ef7badcde","created_at":"2021-11-22T12:00:07.720Z","meta.caller_id":"Cronjob","correlation_id":"d587da0be9acf30a9d8317b13da780df","idempotency_key":"resque:gitlab:duplicate:cronjob:expire_build_artifacts:7e864f760219ae33143ba929fea73d2c9117919da80b189f87db84994980be48","enqueued_at":"2021-11-22T12:00:07.721Z","job_size_bytes":2,"pid":1644679,"message":"ExpireBuildArtifactsWorker JID-1a39b76097110c3ef7badcde: start","job_status":"start","scheduling_latency_s":0.113121}
{"severity":"INFO","time":"2021-11-22T12:00:07.951Z","class":"Namespaces::ScheduleAggregationWorker","args":["113"],"retry":3,"queue":"update_namespace_statistics:namespaces_schedule_aggregation","version":0,"queue_namespace":"update_namespace_statistics","jid":"de3e0a811e407584784417ca","created_at":"2021-11-22T12:00:07.948Z","meta.caller_id":"ExpireBuildArtifactsWorker","meta.feature_category":"continuous_integration","meta.client_id":"ip/","correlation_id":"d587da0be9acf30a9d8317b13da780df","idempotency_key":"resque:gitlab:duplicate:update_namespace_statistics:namespaces_schedule_aggregation:a48ed0441003298938285446a1beea7d3de501a28f4d771fef87bc1ae9c7ddc1","enqueued_at":"2021-11-22T12:00:07.949Z","job_size_bytes":5,"pid":1644679,"message":"Namespaces::ScheduleAggregationWorker JID-de3e0a811e407584784417ca: start","job_status":"start","scheduling_latency_s":0.001676}
{"severity":"INFO","time":"2021-11-22T12:00:07.961Z","class":"Namespaces::ScheduleAggregationWorker","args":["113"],"retry":3,"queue":"update_namespace_statistics:namespaces_schedule_aggregation","version":0,"queue_namespace":"update_namespace_statistics","jid":"de3e0a811e407584784417ca","created_at":"2021-11-22T12:00:07.948Z","meta.caller_id":"ExpireBuildArtifactsWorker","meta.feature_category":"continuous_integration","meta.client_id":"ip/","correlation_id":"d587da0be9acf30a9d8317b13da780df","idempotency_key":"resque:gitlab:duplicate:update_namespace_statistics:namespaces_schedule_aggregation:a48ed0441003298938285446a1beea7d3de501a28f4d771fef87bc1ae9c7ddc1","enqueued_at":"2021-11-22T12:00:07.949Z","job_size_bytes":5,"pid":1644679,"message":"Namespaces::ScheduleAggregationWorker JID-de3e0a811e407584784417ca: done: 0.010041 sec","job_status":"done","scheduling_latency_s":0.001676,"redis_calls":2,"redis_duration_s":0.000381,"redis_read_bytes":2,"redis_write_bytes":228,"redis_queues_calls":2,"redis_queues_duration_s":0.000381,"redis_queues_read_bytes":2,"redis_queues_write_bytes":228,"db_count":2,"db_write_count":0,"db_cached_count":0,"cpu_s":0.004016,"mem_objects":998,"mem_bytes":34056,"mem_mallocs":141,"mem_total_bytes":73976,"duration_s":0.010041,"completed_at":"2021-11-22T12:00:07.961Z","db_duration_s":0.004639}
{"severity":"INFO","time":"2021-11-22T12:00:09.807Z","queue":"cronjob:expire_build_artifacts","args":[],"class":"ExpireBuildArtifactsWorker","retry":0,"version":0,"queue_namespace":"cronjob","jid":"1a39b76097110c3ef7badcde","created_at":"2021-11-22T12:00:07.720Z","meta.caller_id":"Cronjob","correlation_id":"d587da0be9acf30a9d8317b13da780df","idempotency_key":"resque:gitlab:duplicate:cronjob:expire_build_artifacts:7e864f760219ae33143ba929fea73d2c9117919da80b189f87db84994980be48","enqueued_at":"2021-11-22T12:00:07.721Z","job_size_bytes":2,"pid":1644679,"message":"ExpireBuildArtifactsWorker JID-1a39b76097110c3ef7badcde: done: 1.972892 sec","job_status":"done","scheduling_latency_s":0.113121,"redis_calls":6,"redis_duration_s":0.006063000000000001,"redis_read_bytes":265,"redis_write_bytes":1822,"redis_cache_calls":1,"redis_cache_duration_s":0.000126,"redis_cache_read_bytes":203,"redis_cache_write_bytes":62,"redis_queues_calls":3,"redis_queues_duration_s":0.005629,"redis_queues_read_bytes":60,"redis_queues_write_bytes":1456,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.000308,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":304,"db_count":2009,"db_write_count":5,"db_cached_count":1988,"cpu_s":1.794631,"mem_objects":1135121,"mem_bytes":17260672,"mem_mallocs":254581,"mem_total_bytes":62665512,"extra.expire_build_artifacts_worker.destroyed_job_artifacts_count":2,"duration_s":1.972892,"completed_at":"2021-11-22T12:00:09.807Z","db_duration_s":0.131203}
Output of checks
The orphan cleanup job does not find anything relevant:
lscorcia@MOPAPGITLAB01:~$ sudo gitlab-rake gitlab:cleanup:orphan_job_artifact_files DRY_RUN=false
I, [2021-11-22T12:48:51.036823 #1676957] INFO -- : Looking for orphan job artifacts to clean up
I, [2021-11-22T12:48:51.037377 #1676957] INFO -- : find command: '/bin/ionice -c best-effort find -L /var/opt/gitlab/gitlab-rails/shared/artifacts -mindepth 6 -maxdepth 6 -type d'
I, [2021-11-22T12:48:51.486441 #1676957] INFO -- : Processed 12955 job artifact(s) to find and cleaned 0 orphan(s).
However, if I run the following script in the Rails console, I see a bunch of items that should be expired:
project = Project.find_by_full_path('mypath/myproject')
builds_with_artifacts = project.builds.with_downloadable_artifacts
builds_to_clear = builds_with_artifacts.where("finished_at < ?", 2.week.ago)
builds_to_clear.find_each do |build|
puts build.artifacts_expire_at
end
2021-08-25 18:37:31 UTC
2021-08-25 18:35:05 UTC
2021-09-01 16:47:07 UTC
2021-08-25 20:41:45 UTC
2021-08-25 20:43:59 UTC
2021-09-01 18:54:07 UTC
2021-08-26 21:53:23 UTC
2021-08-26 21:51:33 UTC
2021-09-02 20:03:35 UTC
2021-09-01 15:57:52 UTC
2021-09-01 16:00:44 UTC
2021-09-08 14:11:36 UTC
2021-09-01 17:48:03 UTC
2021-09-01 17:50:27 UTC
2021-09-08 16:01:06 UTC
2021-09-02 14:28:22 UTC
2021-09-02 14:50:46 UTC
2021-09-09 13:00:37 UTC
2021-09-03 15:43:30 UTC
2021-09-03 15:45:52 UTC
2021-09-10 13:56:03 UTC
2021-09-08 20:26:32 UTC
2021-09-08 20:29:04 UTC
2021-09-15 18:38:52 UTC
2021-10-19 23:31:51 UTC
2021-10-19 23:34:44 UTC
2021-10-20 20:35:31 UTC
2021-10-20 20:42:10 UTC
2021-10-20 20:43:55 UTC
2021-10-26 23:32:33 UTC
2021-10-26 23:35:55 UTC
2021-10-26 23:27:55 UTC
However, in the UI those very same builds are marked as No artifacts found
(see attached image) in the Pipelines screen:
This is a report of disk usage for the /var/opt/gitlab/gitlab-rails/shared/artifacts
folder:
# du -a . | sort -n -r | head -n 50
74653360 .
28496036 ./86
28496032 ./86/e5
28496028 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef
25633164 ./8e
25633160 ./8e/61
25633156 ./8e/61/8e612bd1f5d132a339575b8dafb7842c64614e56bcf3d5ab65a0bc4b34329407
5489668 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef/2021_08_25
5485552 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef/2021_09_01
4207000 ./8c
4206996 ./8c/d2
4206992 ./8c/d2/8cd2510271575d8430c05368315a87b9c4784c7389a47496080c1e615a2a00b6
2762496 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef/2021_09_02
2758380 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef/2021_09_08
2754876 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef/2021_09_03
2743088 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef/2021_08_26
1984740 ./8e/61/8e612bd1f5d132a339575b8dafb7842c64614e56bcf3d5ab65a0bc4b34329407/2021_11_12
1751148 ./4a
1751144 ./4a/85
1751140 ./4a/85/4a8596a7790b5ca9e067da401c018b3206befbcf95c38121854d1a0158e7678a
1699104 ./73
1690196 ./73/4d
1690192 ./73/4d/734d0759cdb4e0d0a35e4fd73749aee287e4fdcc8648b71a8d6ed591b7d4cb3f
1651024 ./cb
1651020 ./cb/a2
1651016 ./cb/a2/cba28b89eb859497f544956d64cf2ecf29b76fe2ef7175b33ea59e64293a4461
1612408 ./5e
1612404 ./5e/f6
1612400 ./5e/f6/5ef6fdf32513aa7cd11f72beccf132b9224d33f271471fff402742887a171edf
1586532 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef/2021_11_22
1512348 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef/2021_10_26
1501992 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef/2021_10_20
1479536 ./65
1479532 ./65/a6
1479528 ./65/a6/65a699905c02619370bcf9207f5a477c3d67130ca71ec6f750e07fe8d510b084
1400136 ./7b
1400132 ./7b/69
1400128 ./7b/69/7b69759630f869f2723875f873935fed29d2d12b10ef763c1c33b8e0004cb405
1367336 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef/2021_09_08/28367
1367144 ./86/e5/86e50149658661312a9e0b35558d84f6c6d3da797f552a9657fe0558ca40cdef/2021_09_08/28367/43117
Results of GitLab environment info
GitLab Community on-premises setup.
Expand for output related to GitLab environment info
System information System: Ubuntu 20.04 Current User: git Using RVM: no Ruby Version: 2.7.4p191 Gem Version: 3.1.4 Bundler Version:2.1.4 Rake Version: 13.0.6 Redis Version: 6.0.16 Git Version: 2.33.0. Sidekiq Version:6.2.2 Go Version: unknown GitLab information Version: 14.4.2 Revision: 1ce86e92f81 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 12.7 URL: https://gitlab.mydomain.com HTTP Clone URL: https://gitlab.mydomain.com/some-group/some-project.git SSH Clone URL: git@gitlab.mydomain.com:some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers: openid_connect GitLab Shell Version: 13.21.1 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successfulChecking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Checking Reply by email ...
IMAP server credentials are correct? ... Checking gitlab@mydomain.com yes Init.d configured correctly? ... skipped MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 100 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 3/6 ... yes 3/7 ... yes 3/15 ... yes 3/16 ... yes 3/24 ... yes 3/25 ... yes 3/32 ... yes 36/33 ... yes 3/34 ... yes 8/35 ... yes 3/36 ... yes 3/37 ... yes 3/38 ... yes 3/39 ... yes 3/41 ... yes 70/42 ... yes 3/43 ... yes 16/44 ... yes 18/45 ... yes 18/46 ... yes 18/47 ... yes 23/49 ... yes 23/50 ... yes 23/51 ... yes 3/56 ... yes 34/57 ... yes 3/60 ... yes 42/61 ... yes 45/62 ... yes 51/65 ... yes 51/66 ... yes 51/67 ... yes 51/68 ... yes 51/69 ... yes 55/71 ... yes 119/72 ... yes 55/73 ... yes 55/74 ... yes 55/75 ... yes 55/76 ... yes 55/77 ... yes 55/78 ... yes 3/79 ... yes 3/80 ... yes 3/81 ... yes 3/82 ... yes 3/83 ... yes 59/87 ... yes 55/88 ... yes 53/89 ... yes 28/91 ... yes 51/92 ... yes 51/94 ... yes 119/95 ... yes 3/96 ... yes 3/97 ... yes 51/98 ... yes 77/100 ... yes 3/101 ... yes 3/104 ... yes 3/105 ... yes 3/106 ... yes 3/107 ... yes 3/109 ... yes 3/110 ... yes 3/111 ... yes 77/112 ... yes 3/113 ... yes 119/114 ... yes 119/115 ... yes 3/116 ... yes 3/117 ... yes 3/118 ... yes 47/119 ... yes 3/120 ... yes 3/121 ... yes 36/122 ... yes 84/123 ... yes 3/126 ... yes 3/127 ... yes 3/128 ... yes 3/129 ... yes 3/130 ... yes 3/131 ... yes 3/132 ... yes 3/133 ... yes 3/136 ... yes 3/137 ... yes 3/138 ... yes 3/139 ... yes 88/140 ... yes 88/142 ... yes 3/143 ... yes 93/145 ... yes 93/148 ... yes 3/149 ... yes 3/150 ... yes 88/151 ... yes 3/152 ... yes 102/153 ... yes 102/157 ... yes 102/158 ... yes 51/159 ... yes 102/160 ... yes 3/161 ... yes 3/162 ... yes 77/164 ... yes 3/165 ... yes 3/166 ... yes 3/167 ... yes 113/168 ... yes 3/170 ... yes 3/171 ... yes 50/172 ... yes 3/173 ... yes 119/174 ... yes 3/175 ... yes 102/176 ... yes 23/177 ... yes 3/178 ... yes 65/179 ... yes 102/180 ... yes 3/181 ... yes 102/182 ... yes 119/183 ... yes 119/184 ... yes 3/185 ... yes 131/186 ... yes 102/187 ... yes 3/188 ... yes 3/189 ... yes 3/190 ... yes 88/191 ... yes 3/192 ... yes 3/193 ... yes 47/194 ... yes 143/196 ... yes 143/197 ... yes 143/198 ... yes 3/199 ... yes 51/200 ... yes Redis version >= 5.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.4) Git version >= 2.33.0 ? ... yes (2.33.0) Git user has default SSH configuration? ... yes Active users: ... 112 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished