<!--- Please read this! Before opening a new issue, make sure to search for keywords in the issues filtered by the "regression" or "bug" label: - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=bug and verify the issue you're about to submit isn't a duplicate. ---> ### Summary A pipeline which uses stages to make sure deployment can't happen without tests can be tricked. As a consequence, deployments / builds are possible even if tests are failing. In a stage-based pipeline where `deploy-job` is supposed to depend on `test-job` based on the implicit stage dependencies, it's still possible to retry the `deploy-job` if you cancel it while the `test-job` is running. Retrying the `deploy-job` in this case will simply insert it into the running queue disregarding the status of the `test-job`. ### Steps to reproduce 1. Have a pipeline similar to this: ``` test: stage: test script: - # Tests are running here deploy: stage: deploy script: - # Building and deploying to production ``` 2. While the test job runs, cancel the deploy job. 3. Now click "retry" on deploy job 4. The job is directly picked by a runner. - If `dependencies` were set, the job does fail due to missing dependencies. This is ugly but not a dramatic failure - If no dependencies were set, the job just runs trough. This is dramatic **This can not only happen if you are a rouge user but also inadvertently, e.g. you accidentally cancelled a job and then want to have it back in the pipeline.** ### Example Project This CI: https://gitlab.com/PhilLab/testing-private/-/blob/a6514903e34322dc92319fec4fad33e66b3c2259/.gitlab-ci.yml Leads to a successful deployment even though the tests failed: - https://gitlab.com/PhilLab/testing-private/pipelines/120920659 ### What is the current *bug* behavior? Stages which are meant to block subsequent procedures can be jumped over. When a job is restarted, it is not respecting the sequence of stages and waiting to execute only when jobs in the preceding stages have successfully completed. ### What is the expected *correct* behavior? Retrying the job should not directly start it but insert it back into the usual processing queue. Expected behavior as noted in the [docs](https://docs.gitlab.com/ee/ci/pipelines/index.html#cicd-pipelines): - **If _all_ jobs in a stage succeed, the pipeline moves on to the next stage.** - **If _any_ job in a stage fails, the next stage is not (usually) executed and the pipeline ends early** ## Proposal solve the problem in the ~backend by not enqueueing the job directly and use `Ci::ProcessPipelineService` which will look at the whole pipeline. If the `deploy` job meets the dependencies it will run, otherwise it will remain in `created` state until the `test` job completes. https://gitlab.com/gitlab-org/gitlab/-/issues/352858 In https://gitlab.com/gitlab-org/gitlab/-/merge_requests/83730/diffs#note_915332037 we discussed that we should use `Ci::PipelineCreation::StartPipelineService` (to be renamed to `Ci::Pipelines::StartService`) since this service creates a persistent ref (pipeline run prerequisites) and runs `Ci::ProcessPipelineService`. In fact, we should use `StartPipelineService` not only in `PlayBuildService` but also in `RetryJobService` and `RetryPipelineService`. Basically any time we resume the pipeline execution. Once we do that we should also remove the creation of `persistent_ref` in `Ci::Build` state transition since the `persistent_ref` will be consistently created when we start/resume the pipeline execution and deleted when the pipeline completes. **Can the same bug exist also with scheduled jobs (`when: delayed`)?**. If so, we would need to run `StartPipelineService` also in `Ci::RunScheduledBuildService`. <!--- #### User experience specs When the user clicks "Retry" we will show the job in `pending` status until we get the response from BE. If the job cannot be run due to unmet dependencies, we bring the job to its previous status and throw an error. If it can be run, it goes into `running` status. - The alert should appear above the pipeline graph if the job is retried from the pipeline graph, and as the page-level alert on any other page where the job can be retried (job page, pipelines list page, MR page, etc.) - Error alert text: `[job-name] can't be retried because earlier required jobs have not completed successfully yet.` - The error alert should be placed above the pipeline graph and it should be dismissible. ---> ### Output of checks This bug happens on GitLab.com ### Possible related issues I can't link to fixes, but the fix might be in the same realm as some of the other retry bugs: - gitlab-org/gitlab#32712 - gitlab-org/gitlab#37268 - gitlab-org/gitlab#25321 - gitlab-org/gitlab#31367 ### Testing Consideration In terms of testing, let's include a test in the integration level with the FE to test this exact scenario. 1. Create a pipeline with 2 stages. 2. Run the pipeline. 3. Cancel the first stage. 4. Retry the second stage. 5. Validate that the second stage fails and returns an error.