GitLab.com pipeline jobs stuck in `created` status

Summary

Some pipeline jobs on GitLab.com do not move beyond created status (unless cancelled).

Steps to reproduce

We have at present 3 customer tickets (345101, 345206 and 345595. - internal links) reporting this problem.

Unfortunately, so far we have been unable to reproduce the behaviour, however I did observe the same with an internal test project on the 14th Nov PM.

Two of the pipelines from the tickets above still have jobs in a created state.

Example Project

Paths to the affected pipelines in question will be shared in an internal note.

What is the current bug behavior?

It is unclear how the jobs/pipeline enters this state. Observations from customers is simply that the jobs are 'stuck', retrying the jobs does not change the status, they remain created.

What is the expected correct behavior?

The jobs should move from created and progress normally.

Relevant logs and/or screenshots

See internal notes.

Output of checks

This bug happens on GitLab.com

Possible fixes

I believe in most cases the pipeline can be re-run, however we need to determine root cause and be sure this is not an ongoing problem.