Store failed builds in the cache

Description

Changes proposed in this merge request:

• Add metadata for the result of a build.
• Store failed build artifacts in the cache unconditionally.
• Fix up call sites where it is assumed that if it is cached it was successful.
• Skip straight to failing a build if it is a cached fail.
• Permit manual retrying by flagging that it's already skipped the build once so if it's queued again it rebuilds.

This merge request, when approved, will close issue/bug: #76 (closed)

CHANGELOG/Release Notes

• Made it push artifacts. --on-error=continue was only missing putting the artifact in done so it could be queued for push. --on-error=quit was more difficult and may not have an adequate solution.
• Fixed after initial review and tidied the history. #76 (comment 74605228) yet to be addressed.
• Rebased and code tidied. #76 (comment 74605228) yet to be addressed and history tidied.

added 50 commits

• a05c8eec...c0de75e2 - 42 commits from branch master
• 8d37d7d7 - WIP
• 1f5668c5 - WIP: Rebuild failed builds
• b38a2c49 - WIP: skip rebuilding cached failures
• 98645712 - WIP: Fix conditionals to permit old artifacts
• 71f72493 - WIP: failing to skip builds
• bfece875 - WIP: Fail build properly
• 2ad6130f - WIP: Drop debugging print statements
• 8178fccb - WIP: Infer which key to use in get_public_data from context

Compare with previous version

added 1 commit

• 606d306e - WIP: Cache failed builds

Compare with previous version

changed the description

The overall patch looks good to me. I was wondering about one thing that was mentioned in the original issue - --force option to retry the build. Do we plan on providing some kind of force-retry mechanism for failed builds, either separately or as part of this change?

#not-a-maintainer

resolved all discussions

@cs-shadow I was wondering that but I'm not convinced we'll need it. That's one of the things I'm hoping to get some consensus of from review.

You can force-retry a build from the command-line, though automation won't necessarily have that option. If so that automation might be better served by not caching failures instead.

resolved all discussions

added 1 commit

• d49a7c5c - 1 commit from branch master

Compare with previous version

added 6 commits

• 231e4a13 - WIP: Cache the result of checking whether an artifact is cached weakly
• 89ae6aa0 - WIP: Add metadata to distinguish between successful and failed builds
• 1c4a5663 - WIP: Convert call-sites of _cached that assume success
• 45167821 - WIP: Render cached failures differently to successes
• d93c626b - WIP: Cache failed builds too
• cb42bca9 - WIP: Skip rebuilding cached failures

Compare with previous version

resolved all discussions

added 1 commit

• 8a820a2d - WIP: Add unit tests for cached behaviours

Compare with previous version

resolved all discussions

added 8 commits

• 7694d190 - element.py: Cache the result of checking whether an artifact is cached weakly
• 7ac61251 - element.py: Add metadata to distinguish between successful and failed builds
• 06277b23 - Convert call-sites of Element._cached() that assume success
• 76e08eda - _frontend/widget.py: Render cached failures differently to successes
• 74b7bf63 - Cache failed builds
• 06db48dd - _scheduler/queue.py: Skip rebuilding cached failures
• 6e67d5a8 - tests: Add unit tests for cached behaviours
• b66e8dd0 - NEWS: Describe caching of failures

Compare with previous version

unmarked as a Work In Progress

changed the description

@cs-shadow, @richardmaw-codethink: Regarding the pondered --force flag

I think a semantic might be needed, but think we can handle that separately.

The point I've been driving at in #76 (closed) is that the biggest win we get here is the ability to debug builds which happened somewhere else; implementing this completely will probably have to depend on solving #21 (closed) (by way of #311) first, though.

The behavior I think I would ultimately expect would be:

• When the failed element reaches the build queue in interactive mode:
  • Construct a nice log line, including traceback of the failed build log
  • Offer the user the regular choices, the user can now debug the failed build in a shell
• When in non interactive mode:
  • Follow the --on-error policy (decided by command line or user config file)
• This would be a change from the current behavior:
  • When building something which previously failed, we would now jump right to the prompt instead of retrying from scratch
  • We might also offer a new option to ignore failed artifacts and retry them over (or delete it and try again ?).

What a --force flag would potentially allow, would be the way to ignore existing failed artifacts and retry them from scratch instead of using their preserved failed state - I'm not sure how handy this will be (because changing the input to try something new, will make a new cache key, consequently a whole new artifact) - but it might turn out to be nice to have.