1000+ pending QA jobs

Summary

There are over 1000 jobs pending in the GitLab QA mirror (and some in GitLab QA too).

It's not clear what impact that has, but it's likely to mean waiting much longer for results than necessary.

Questions

1. Where are all the jobs coming from? Are they stale jobs, or are they all for recent MRs or scheduled pipelines?
   • They were for recent MRs with feature flag definition changes
2. Can we reduce the number of jobs?
   • Done: gitlab-org/gitlab-qa!775 (merged)
   • The jobs are now manual
3. Should we increase the size of the runner, or add a new one?
   • We should increase the number of jobs the runner can handle once we have ownership of the runner.

Background

Slack discussions:

• https://gitlab.slack.com/archives/CB3LSMEJV/p1634715515319500
• https://gitlab.slack.com/archives/C3JJET4Q6/p1634690918049600

The relevant runner: (the same runner in both projects)

• https://gitlab.com/gitlab-org/gitlab-qa/-/runners/207549
• https://gitlab.com/gitlab-org/gitlab-qa-mirror/-/runners/207549

Runner config: https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/blob/master/roles/build-trigger-runner-manager-gitlab-org.json