Be smarter about querying summaries from remote caches
Currently at the start of a build we download the summary file for the configured remote cache once, and we don't update it.
Once !166 (merged) is merged we will query the summary file for every configured remote cache once at the start of the build.
There are two use cases where this isn't ideal.
Co-operative builds
If two machines which share a remote cache are building a similar set of elements, they will not be aware of each other's progress. Each downloads the summary file for the cache once and then does not update it.
If both machines update their copies of the summary file regularly during the build then they may be able to share artifacts more effectively, for example if machine A built GLIBC before machine B needed GLIBC, machine B could use the prebuilt version of GLIBC instead of running its own build.
Local mirroring
A project may have a remote artifact cache set up in some distant network where it takes me a long time to connect. To mitigate this I might set up a local mirror which syncs from the remote cache in the background. Assuming !166 (merged) in its present state is merged, I will be able to add this new cache as higher priority than the project-specific one and BuildStream will prefer to download artifacts from the local cache. However, I will still have to wait for the summary file from the distant slow cache to be downloaded at the start of every build -- this isn't ideal and could be fixed if we avoided contacting the slow cache until we actually need to (for example if we need an artifact that isn't in the mirror).
This second use case isn't a real world problem yet, I don't know of anyone actually doing this kind of mirroring.