Artifact As A Proto
Background
Umbrella ticket (aka, epic) to capture the tasks required to implement Artifact As A Proto. See also the proto and gRPC service outline on the list.
Task description
-
1) Create abstractions in BuildStream (without tests) - #908 (closed) and #955 (closed) - Goals: Have abstracted anything in
Element
which deals with the internals of an artifact intoArtifact
- result should not necessarily need additional tests (though more are welcome) since everything should already be covered - Results: There exists an
Artifact
class which is {1,2}:1 withElement
for now (i.e. anElement
has 1 or 2Artifact
instances representing the strong and weak artifacts as appropriate. No graph is present withinArtifact
- Goals: Have abstracted anything in
-
2) Create abstractions in testutils - #945 (closed) - Goals: Have abstracted anything within the tests which deals with the internals of an artifact into a
TestArtifact
class. This should not need additional tests, and should leave us with all tests still passing. - Results: There exists a
TestArtifact
class in thetestutils
which allows tests to talk about the semantics of the internals of artifacts without actually caring about how they are represented on disk. Caveat: If a test's purpose is to actually verify the physical layout then we'll need to keep that short-term but expect the test to be replaced during subsequent work.
- Goals: Have abstracted anything within the tests which deals with the internals of an artifact into a
NB the above two tasks can be worked on in parallel
-
3) Review abstractions - no specific ticket, but this is considered complete and closed - Goals: Ensure that the abstractions created in 1 and 2 are consistent and that there do not remain any un-abstracted assumptions about artifact layout in any of
Element
or the tests (modulo the above caveat). - Results: Any missing abstractions are added, we're ready to proceed.
- Goals: Ensure that the abstractions created in 1 and 2 are consistent and that there do not remain any un-abstracted assumptions about artifact layout in any of
-
4) Create ArtifactService protos, gRPC service - #965 (closed) - Goals: The artifact server service protos and gRPC servicer instances should be created, with a view to providing the basis on which the reworked buildstream code can be tested.
- Results:
bst-artifact-server
supports both the old and the new artifact services so that it can be merged short-term until cleaned up when the next job finishes.
-
5) Re-work artifact and test artifact class to use the artifact proto - #974 (closed) - Goals: All existing physical artifact layout assumptions are removed, replaced with the artifact as a proto proto. Use of
ReferenceService
is removed and replaced with use ofArtifactService
along with commensurate support for the optionalities expressed in that. - Results: The bulk of the work is now complete, we're using AaaP and issues will shake out.
- Goals: All existing physical artifact layout assumptions are removed, replaced with the artifact as a proto proto. Use of
* [ ] 6) Reference Service Excision
* Goals: Remove the Reference Service from the code entirely, cease supporting it, cease walking old style refs for mtime updates in the
CASCache
etc.* Results: We no longer have the duality present in bst-artifact-server
mentioned in 4. Caveat: It may be worthwhile supporting both on the server-side for a while longer, for migration reasons - perhaps discuss at the time?
Note: the above point was edited to strike through the text as we agreed to continue to support the reference service (see comments below).
-
7) Capabilities Service - Goals: Creation of a proto and gRPC service to represent the capabilities of
bst-artifact-server
- Results: We can query an artifact service to understand if it will accept build trees, what its policy is on storing failed results or log files, or whatever else we want to allow configuration of.
- Goals: Creation of a proto and gRPC service to represent the capabilities of
NB the above is to be deferred and done separately to this undertaking (ie, when it's actually needed) See #915 (closed)
-
8) Verify remote execution behaviour: ensure it copes without the old reference storage service - Goals: Ensure that remote execution continues to function, even without reference storage
- Results: We will be confident that this work has not resulted in RE failing. This may go hand in hand with the work on 6, depending on whether or not they are intertwined.
NB the above task to be done throughout the process, not just at the end