## Background Umbrella ticket (aka, epic) to capture the tasks required to implement [Artifact As A Proto](https://mail.gnome.org/archives/buildstream-list/2018-December/msg00148.html). See also the [proto and gRPC service outline](https://mail.gnome.org/archives/buildstream-list/2019-February/msg00027.html) on the list. ## Task description * [x] 1) Create abstractions in BuildStream (without tests) - https://gitlab.com/BuildStream/buildstream/issues/908 and https://gitlab.com/BuildStream/buildstream/issues/955 * Goals: Have abstracted anything in `Element` which deals with the internals of an artifact into `Artifact` - result should not necessarily need additional tests (though more are welcome) since everything should already be covered * Results: There exists an `Artifact` class which is {1,2}:1 with `Element` for now (i.e. an `Element` _has_ 1 or 2 `Artifact` instances representing the strong and weak artifacts as appropriate. No graph is present within `Artifact` * [x] 2) Create abstractions in testutils - https://gitlab.com/BuildStream/buildstream/issues/945 * Goals: Have abstracted anything within the tests which deals with the internals of an artifact into a `TestArtifact` class. This should not need additional tests, and should leave us with all tests still passing. * Results: There exists a `TestArtifact` class in the `testutils` which allows tests to talk about the semantics of the internals of artifacts without actually caring about how they are represented on disk. __Caveat__: If a test's purpose is to actually verify the physical layout then we'll need to keep that short-term but expect the test to be replaced during subsequent work. **NB the above two tasks can be worked on in parallel** * [x] 3) Review abstractions - no specific ticket, but this is considered complete and closed * Goals: Ensure that the abstractions created in _1_ and _2_ are consistent and that there do not remain any un-abstracted assumptions about artifact layout in any of `Element` or the tests (modulo the above caveat). * Results: Any missing abstractions are added, we're ready to proceed. * [x] 4) Create ArtifactService protos, gRPC service - https://gitlab.com/BuildStream/buildstream/issues/965 * Goals: The artifact server service protos and gRPC servicer instances should be created, with a view to providing the basis on which the reworked buildstream code can be tested. * Results: `bst-artifact-server` supports both the old and the new artifact services so that it can be merged short-term until cleaned up when the next job finishes. * [x] 5) Re-work artifact and test artifact class to use the artifact proto - https://gitlab.com/BuildStream/buildstream/issues/974 * Goals: All existing physical artifact layout assumptions are removed, replaced with the artifact as a proto proto. Use of `ReferenceService` is removed and replaced with use of `ArtifactService` along with commensurate support for the optionalities expressed in that. * Results: The bulk of the work is now complete, we're using AaaP and issues will shake out. ~~* [ ] 6) Reference Service Excision~~ ~~* Goals: Remove the Reference Service from the code entirely, cease supporting it, cease walking old style refs for mtime updates in the `CASCache` etc.~~ ~~* Results: We no longer have the duality present in `bst-artifact-server` mentioned in 4. __Caveat__: It may be worthwhile supporting both on the server-side for a while longer, for migration reasons - perhaps discuss at the time?~~ **Note:** the above point was edited to strike through the text as we agreed to continue to support the reference service (see comments below). * [ ] 7) Capabilities Service * Goals: Creation of a proto and gRPC service to represent the capabilities of `bst-artifact-server` * Results: We can query an artifact service to understand if it will accept build trees, what its policy is on storing failed results or log files, or whatever else we want to allow configuration of. **NB the above is to be deferred and done separately to this undertaking (ie, when it's actually needed)** See https://gitlab.com/BuildStream/buildstream/issues/915 * [ ] 8) Verify remote execution behaviour: ensure it copes without the old reference storage service * Goals: Ensure that remote execution continues to function, even without reference storage * Results: We will be confident that this work has not resulted in RE failing. This *may* go hand in hand with the work on 6, depending on whether or not they are intertwined. **NB the above task to be done throughout the process, not just at the end**