Add a way to retry failed CI jobs
We are often seeing issues where a job fails, but not due to the source that we are building.
For example, many times we've seen issues where GitLab fails to access another GitLab repository (Could not resolve host: gitlab.com). For example: https://gitlab.com/cip-playground/linux-stable-rc-ci/-/jobs/383851876.
In other cases jobs have failed due to "error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500 Internal Server Error".
We've also seen problems with uploading artifacts: https://gitlab.com/cip-playground/linux-stable-rc-ci/-/jobs/380700161
Until GitLab becomes more reliable, perhaps the workaround is to just automatically retry jobs using retry
: https://docs.gitlab.com/ee/ci/yaml/#retry
We should only retry for infrastructure issues though, not from when errors are detected in the source or during testing on the target.