Users can retry cancelled jobs in further stages of a pipeline

Summary

Marking as a gitlab-ce2278648, but probably it can be considered also an gitlab-ce2975007.

If I run a pipeline, I cannot force jobs that are "later" in the list to run earlier. They have to wait until all the previous jobs succeeds (with some exception).

But if I cancel the pipeline immediately, they I can run (by retrying) any job in the pipeline without respecting the original order.

Maybe it is not so bad, maybe it can be useful for some edge cases, but it is inconsistent with the "standard" flow.

Steps to reproduce

1. Create a pipeline with more than one stage
2. Cancel it
3. Retry a cancelled job in the second stage

Example Project

https://gitlab.com/bikebilly/demo-dep/pipelines/12200636

What is the current bug behavior?

You can retry cancelled jobs in any order.

What is the expected correct behavior?

Stage order should be respected.

Relevant logs and/or screenshots

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:env:info)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)