Add logic to CI config to guard against site outages due to artifact upload errors
Description
We should guard against the possibility of a site outage like the one in gitlab-com/gl-infra/production#2088 (closed), because the underlying issue looks like it will not have a fix or workaround soon
Another type of problem that causes this is when the underlying device is full: https://gitlab.com/gitlab-com/www-gitlab-com/-/jobs/601009687#L22
The fix should be an approach similar to 69803355 (but for real files, not symlinks), and abort the master deploy job if any of the nine current build_master jobs failed to successfully upload their artifacts.
Related: Ideally, the build should fail if any of the artifacts downloads fail. This issue relates to that problem: gitlab-org/gitlab#212349 (closed).
Specifically, this comment asks if an issue has been opened for that:
if an artifact download is not successful, for any reason, the job should fail and not attempt to run any scripts.
@tmaczukinor@stanhu- do either of you know if an issue ever get opened on the GitLab Rails side to address that? I don't see one linked on this issue.