CAS server doesn't seem to automatically remove artifacts so it gets fill up
What is the current bug behavior?
This is a continuation of #136 (closed):
Even with the new cache server implementation (CAS), the cache server gets fill up and It doesn't clean the artifacts automatically
Bugs in downstream projects:
- gnome-build-meta: https://gitlab.gnome.org/GNOME/gnome-build-meta/issues/48
- freedesktop-sdk: freedesktop-sdk/freedesktop-sdk#393 (closed)
What is the expected correct behavior?
The server should automatically remove artifacts to have enough space
Relevant logs and/or screenshots
https://gitlab.gnome.org/GNOME/gnome-build-meta/-/jobs/92168
From the journal on the server (not messages at all from bst-artifact-server before that):
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: Exception calling application: Failed to hash object: [Errno 28] No space left on device
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: Traceback (most recent call last):
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: File "/usr/local/lib/python3.6/site-packages/buildstream/_artifactcache/cascache.py", line 398, in add_object
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: out.write(chunk)
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: File "/usr/lib64/python3.6/tempfile.py", line 483, in func_wrapper
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: return func(*args, **kwargs)
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: OSError: [Errno 28] No space left on device
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: During handling of the above exception, another exception occurred:
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: Traceback (most recent call last):
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: File "/usr/local/lib/python3.6/site-packages/buildstream/_artifactcache/cascache.py", line 411, in add_object
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: os.link(out.name, objpath)
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: File "/usr/lib64/python3.6/tempfile.py", line 501, in __exit__
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: result = self.file.__exit__(exc, value, tb)
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: OSError: [Errno 28] No space left on device
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: The above exception was the direct cause of the following exception:
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: Traceback (most recent call last):
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: File "/usr/local/lib64/python3.6/site-packages/grpc/_server.py", line 389, in _call_behavior
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: return behavior(argument, context), True
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: File "/usr/local/lib/python3.6/site-packages/buildstream/_artifactcache/casserver.py", line 186, in Write
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: digest = self.cas.add_object(path=out.name)
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: File "/usr/local/lib/python3.6/site-packages/buildstream/_artifactcache/cascache.py", line 418, in add_object
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: raise ArtifactError("Failed to hash object: {}".format(e)) from e
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: buildstream._exceptions.ArtifactError: Failed to hash object: [Errno 28] No space left on device
Other relevant information
- BuildStream version affected: 1.1.7 (on the server)
Edited by Javier Jardón