Manual "on_failure" type job with needs relationship to prior manual (allowed to fail) job is skipped as are all stages

Searched for existing issues using the type::bug tag and didn't find anything, so here goes.

Summary

After reading the docs about the needs keyword and this GitLab blog post I assumed I could set up a non-blocking manual job that has a dependent blocking manual job to define a pipeline that:

  1. has an entirely automated build pipeline when all jobs succeed, and
  2. has a pipeline with a single blocking manual job when a job inside a previous stage fails

I assumed this because the needs documentation states:

Use needs to execute jobs out-of-order. Relationships between jobs that use needs can be visualized as a directed acyclic graph. You can ignore stage ordering and run some jobs without waiting for others to complete. Jobs in multiple stages can run concurrently.

And the blog post detailing the changes to manual job behavior states:

Now, if there is a needs relationship pointing to a manual job, the pipeline doesn't stop by default anymore. The manual job is considered optional by default in all cases now. Any jobs that have a needs relationship to manual jobs are now also considered optional and skipped if the manual job isn't triggered. If you start the manual job, the jobs that need it can start after it completes.

I thought marking a job with a needs relationship to a manual (non-blocking) job earlier in the pipeline as allow_failure: false would ensure it isn't skipped despite it's parent (the job it needs) being allowed to fail.

Terms

  • blocking = allow_failure: false
  • non-blocking = allow_failure: true
  • automatic = all non-manual jobs

Scenario A stated using IFTTT terminology:

  • IF all non-blocking, automated jobs in each stage succeed
  • AND there is a non-blocking manual job in stage 1
  • THEN the pipeline continues chronologically without stopping
  • THAT causes the entire pipeline to be marked as a success

Scenario A works as expected.

Scenario B states using IFTTT terminology (both scenarios are within the same pipeline):

  • IF any automatic job during stage 1 fails
  • AND there is a matching on_failure rule for a job in stage 2
  • AND the stage 2 job is blocking
  • AND the stage 2 job defines a needs relationship to a non-blocking, manual stage 1 job
  • THEN the stage 2 job should pause the pipeline
  • THEN running the non-blocking, manual stage 1 job should cause the dependent, blocking stage 2 job to execute
  • THEN this should cause all jobs in later stages to be run (if all jobs in stage 2 succeed)
  • THAT causes the entire pipeline to be marked as a $(configurable success)

The configurable success is more of a feature request - but right now - it seems like this bug is due to some contra-indicative documentation terms. The stages docs state all sequential stages will be skipped if any job in a previous stage fails. The needs docs state jobs can be run out of order and that you can "... ignore stage ordering". The relationship between how these two things interact is not clear.

I would like to be able to specify that:

  1. pipeline success can be configured to be based on a specific stage or combination of specific stages succeeding (instead of the default all)
  2. stages can be configured to always run if a specific job or combination of jobs from previous stages succeed (instead of the default all non-blocking jobs from all previous stages)

This would serve a different function to how needs relationships work as it would allow dev-ops engineers that don't want to manage acyclic relationships between jobs configure what jobs are considered necessary for a deploy, which is again different to forcing a zero exit code as allow_failure does.

Steps to reproduce

Given the following configuration:

stages:
  - one
  - two
  - three
  - four
  - five

job1:
  stage: one
  script:
    - echo "build"

job2.0:
  stage: two
  script:
    - echo "job that sometimes fails"

job2.1:
  stage: two
  script:
    - echo "job used to ignore stage two failures, allow_failure is true by default but listing here for clarity"
  when: manual
  allow_failure: true
   
job3.0:
  stage: three
  script: echo "runs only if a job from stage 2 fails and pauses execution of the pipeline as it is marked as manual non-optional"
  when: on_failure
  needs:
    - job: job2.1
      allow_failure: false

job3.1:
  stage: three
  script: echo "runs only if all jobs from stage 2 succeed"
  when: on_success

job4.0:
  stage: four
  script: echo "job 4.0 -- should not be skipped if either job 3.0 or 3.1 succeed"
  when: on_success

job5.0:
  stage: five
  script: echo "job 5.0 -- should not be skipped if job 4.0 succeeds"
  when: on_success

This causes a manual job2.1 to appear, which upon completion causes job3.0 to start (if job2.0 fails), however all remaining stages and jobs are still marked as skipped. So job4.0 and job5.0 are never run.

What is the current bug behavior?

There is no way to tell configure consequent stages not to be skipped and it's unclear, from the docs, what the behavior of consecutive jobs and stages will be. There also doesn't seem to be a way to mark a job as allowed to fail and still define a later job that depends on the real exit code of the script without passing variables around between jobs.

What is the expected correct behavior?

Configurable stage and pipeline success/skip management.

Edited by Che Fisher