CAS server doesn't seem to automatically remove artifacts so it gets fill up

What is the current bug behavior?

This is a continuation of #136 (closed):

Even with the new cache server implementation (CAS), the cache server gets fill up and It doesn't clean the artifacts automatically

Bugs in downstream projects:

• gnome-build-meta: https://gitlab.gnome.org/GNOME/gnome-build-meta/issues/48
• freedesktop-sdk: freedesktop-sdk/freedesktop-sdk#393 (closed)

What is the expected correct behavior?

The server should automatically remove artifacts to have enough space

Relevant logs and/or screenshots

https://gitlab.gnome.org/GNOME/gnome-build-meta/-/jobs/92168

From the journal on the server (not messages at all from bst-artifact-server before that):

Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: Exception calling application: Failed to hash object: [Errno 28] No space left on device
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: Traceback (most recent call last):
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]:   File "/usr/local/lib/python3.6/site-packages/buildstream/_artifactcache/cascache.py", line 398, in add_object
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]:     out.write(chunk)
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]:   File "/usr/lib64/python3.6/tempfile.py", line 483, in func_wrapper
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]:     return func(*args, **kwargs)
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: OSError: [Errno 28] No space left on device
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: During handling of the above exception, another exception occurred:
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: Traceback (most recent call last):
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]:   File "/usr/local/lib/python3.6/site-packages/buildstream/_artifactcache/cascache.py", line 411, in add_object
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]:     os.link(out.name, objpath)
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]:   File "/usr/lib64/python3.6/tempfile.py", line 501, in __exit__
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]:     result = self.file.__exit__(exc, value, tb)
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: OSError: [Errno 28] No space left on device
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: The above exception was the direct cause of the following exception:
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: Traceback (most recent call last):
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]:   File "/usr/local/lib64/python3.6/site-packages/grpc/_server.py", line 389, in _call_behavior
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]:     return behavior(argument, context), True
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]:   File "/usr/local/lib/python3.6/site-packages/buildstream/_artifactcache/casserver.py", line 186, in Write
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]:     digest = self.cas.add_object(path=out.name)
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]:   File "/usr/local/lib/python3.6/site-packages/buildstream/_artifactcache/cascache.py", line 418, in add_object
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]:     raise ArtifactError("Failed to hash object: {}".format(e)) from e
Aug 17 21:30:52 cache-server-fedora-s-2vcpu-2gb-ams3-01 bst-artifact-server[1067]: buildstream._exceptions.ArtifactError: Failed to hash object: [Errno 28] No space left on device

Other relevant information

• BuildStream version affected: 1.1.7 (on the server)