Skip to content

Injected jobs don't make the pipeline fail if next jobs are not specifically using needs: keyword pointing to them

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Steps to reproduce

  1. To reproduce this issue you should have a .gitlab/security-policies/policy.yml with the following content in your security policy project:
---
scan_execution_policy:
- name: This Policy injects Jobs in other projects pipelines
  description: This policy enforces pipeline configuration to have Jobs that run for all branches
  enabled: true
  pipeline_config_strategy: override_project_ci
  rules:
  - type: pipeline
    branches:
    - '*'
  actions:
  - scan: custom
    ci_configuration: |-
      test-job-from-policy:
        stage: .pipeline-policy-test
        script:
          - echo "Running mandatory security scan..."
          - echo "If this job fail, the pipeline should stop and not proceed to next stages/jobs even if those use needs keyword."
          - exit 1            # make it fail on purpose to test the reproduction

The configuration above injects a job called test-job-from-policy into the stage .pipeline-policy-test. We are assuming that you have the feature "Pipeline execution policies" enabled so this works. This test Mimics that a Security Scan is running and that it fails, exiting with 1 to make the pipeline fail; then, the pipeline must not proceed to the next stages as it fail in the security scan, even if the next jobs use needs on other jobs.

  1. Next you must set this security policy project in a group or repository.
  2. The repository must have a .gitlab-ci.yml with jobs that use needs keywords; for reproducing purposes, you can use this:
stages:
  - build
  - test
  - deploy

build-job:
  stage: build
  script:
    - echo "building... done"

test-job:
  stage: test
  script:
    - env | sort | tee -a artifact.txt
  artifacts:
    paths:
      - artifact.txt

deploy-job:
  stage: deploy
  script:
    - echo "deploying... done"
  needs:
    - test-job
  dependencies:
    - test-job

As the pipeline above contains the test stage, according to the documentation, the job test-job-from-policy will be placed into the stage .pipeline-policy-test after the test stage; that means that the stage will be placed between test and deploy stages, like this example:

stages:
  - build
  - test
  - .pipeline-policy-test # this was added by the pipeline execution policy configuration
  - deploy

But notice the job deploy-job: use needs: keyword to need on test-job.

  1. As the pipeline runs, the job test-job-from-policy: fails but the pipeline proceed executing the job deploy-job while the desired behaviour is to Stop the pipeline. image

Desired Behaviour

As a Policy Team Member of a Large Organisation, which has hundreds or thousands of projects repositories, I want to use pipeline execution policies to inject jobs (i.e. enforce security scan jobs) into other projects pipelines so that if an injected job fails, the pipeline must stop and fail.

Is useless for a security team that manages a large organisation, which has hundreds or thousands of projects, to inject a job that might fail but the pipeline still executing.

Using .pre stage is not a solution, as it could be too early to do i.e. a security scan, as a team can be downloading external software code in further stages of the pipeline.

A job injected by a policy should take priority/precedence over other jobs from the pipeline, so if it fails, the pipeline also fails; otherwise it is just a job inserted in the pipeline.

Workarounds (not good)

  • As a workaround, we can modify the pipeline to update the needs: keyword so it also needs the test-job-from-policy; but when you have hundreds or thousands of projects/pipelines, this requires a lot of effort. Definitively not good
  • As another workaround, we can modify the Pipeline execution policy to add an "overwrite" in the "deploy-job" needs: so it also includes the test-job-from-policy; but if you have other pipelines that does not contain a job named "deploy-job", it causes more trouble.

I think that the optimal solution is, if some job fail on .pipeline-policy-test stage, the pipeline should not proceed.

BR, Saul

Edited by 🤖 GitLab Bot 🤖