Store failed builds in the cache

Description

Changes proposed in this merge request:

• Add metadata for the result of a build.
• Store failed build artifacts in the cache unconditionally.
• Fix up call sites where it is assumed that if it is cached it was successful.
• Skip straight to failing a build if it is a cached fail.
• Permit manual retrying by flagging that it's already skipped the build once so if it's queued again it rebuilds.

This merge request, when approved, will close issue/bug: #76 (closed)

CHANGELOG/Release Notes

• Made it push artifacts. --on-error=continue was only missing putting the artifact in done so it could be queued for push. --on-error=quit was more difficult and may not have an adequate solution.
• Fixed after initial review and tidied the history. #76 (comment 74605228) yet to be addressed.
• Rebased and code tidied. #76 (comment 74605228) yet to be addressed and history tidied.

buildstream/_scheduler/queue.py

@@ -21,6 +21,7 @@
 
 # System imports
 from collections import deque
+from datetime import timedelta
 from enum import Enum
 import traceback
 
@@ -91,6 +92,7 @@ class Queue():
         self._max_retries = 0
         if self.queue_type == QueueType.FETCH or self.queue_type == QueueType.PUSH:
             self._max_retries = scheduler.context.sched_network_retries
+        self._tried = set()
 
         # Assert the subclass has setup class data
         assert self.action_name is not None
@@ -240,6 +242,19 @@ class Queue():
 
             self.prepare(element)
 
+            if element._cached_failure() and element not in self._tried:
+                result_data = element.get_public_data("result")
+                assert "description" in result_data and "detail" in result_data
+                self._message(element, MessageType.FAIL, result_data["description"],
+                              detail=result_data["detail"], action_name=self.action_name,
+                              elapsed=timedelta(seconds=0))
+                self._scheduler.put_job_token(self.queue_type)
+                self._done_queue.append(element)
+                self.failed_elements.append(element)
+                self._scheduler._job_complete_callback(element, self, self.action_name, False)
+                self._tried.add(element)
+                continue
+
             job = Job(scheduler, element, self.action_name,
                       self.process, self._job_done,
                       max_retries=self._max_retries)