Allow ContentAddressableStorage and ByteStream to be read only
Description
In the case of using BuildGrid only for a remote cache (that is CAS, optionally with an action cache or reference store), then it may be desirable for the CAS to be read only. For a motivating example, consider a BuildStream project that wants to have a public cache for pulling so that anyone can download the artifacts. The way I have usually seen this done is to run a push server and a pull server with some form of mirroring. However, in order for this to be secure the CAS needs to be read only, as otherwise anyone could just throw blobs at it.
Changes proposed in this merge request:
- Allow CAS to be read only
Validation
I've managed to validate this patch worked correctly. To test this I used the following docker-compose.yml
:
version: '3'
services:
buildgrid-pull:
image: registry.gitlab.com/coldtom/coldtom-dockerfiles/bgd:ro-cas
ports:
- '5501:5501'
expose:
- '5501'
volumes:
- "./config:/home/bgd/config"
- "./bstcache/bgd:/data"
entrypoint:
- bgd
- server
- start
- --verbose
- /home/bgd/config/storage-pull.conf
bst-artifact-push:
image: buildstream/buildstream:nightly
ports:
- "1102:1102"
expose:
- '1102'
volumes:
- "./bstcache/bst:/artifacts"
entrypoint:
- bst-artifact-server
- --enable-push
- -p 1102
- --index-only
- /artifacts
buildstream:
image: buildstream/buildstream:nightly
volumes:
- "./test:/test"
links:
- buildgrid-pull
- bst-artifact-push
privileged: true
stdin_open: true
tty: true
command:
- "/bin/bash"
The storage-pull.conf looks like so:
server:
- !channel
port: 5501
insecure-mode: true
description:
Shared Bazel/BuildStream CAS (pull only) configuration:
- Unauthed plain HTTP at :5501
- On-disk data stored in /data
- Hosted services:
- ActionCache
- ContentAddressableStorage
- ByteStream
authorization:
method: none
monitoring:
enabled: false
instances:
- name: ''
storages:
- !disk-storage &main-storage
path: /data
services:
- !action-cache
storage: *main-storage
max-cached-refs: 256
allow-updates: false
- !cas
storage: *main-storage
read-only: true
- !bytestream
storage: *main-storage
thread-pool-size: 1000
I then tried building a simple import
element and pushing it to the remote. It failed with the correct error code. I could still pull artifacts that were already cached.