CI instabilities getting in the way of pipelines success rate and developer productivity

There are currently a list of problems that make our pipelines unstable, mostly during the "artifacts upload" part of a job (the very last part), roughly ordered by occurrences:

Unresolved

Resolved

  1. 500 error during artifact uploads gitlab-org/gitlab#36512 (closed) => Moved to %12.9
    1. gitlab-org/gitlab#24286 (closed) => %12.7 => Moved to %12.8
    2. gitlab-org/gitlab#36516 (closed) => No milestone => Moved to %12.8
  2. 429 error error during artifact uploads https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8308 / gitlab-org/gitlab-runner#4796 (closed) => Fixed at haproxy level (https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8308#note_268374823) on 2020-01-08 => Done
  3. No space left on device gitlab-org/gitlab#8008 (closed) => %12.8 => Moved to %12.9
  4. Jobs are timing out
  5. Transient failure in build-qa-image while trying to log into the container registry gitlab-org/gitlab#118676 (closed) => %12.7 => Moved to %12.8

I've extracted the list from https://gitlab.com/gitlab-org/quality/team-tasks/issues/256 since this wasn't the main point of the issue.

Potential root causes and solutions

It looks like all issues are infrastructure-related, and I found some infrastructure issues that could explain these:

/cc @gl-quality/eng-prod

Edited by Jason Yavorsky