Backend: Improve the error message when no pipeline is created if jobs with `needs` can't run
**Backend: 'somejob' job needs 'otherjob' job, but 'otherjob' is not in any previous stage** Original description
Summary
Gitlab CI pipeline is unable to find the job in any previous stage but it's there, and it is also show (partially) on the Visualize tab, and the yaml does not look to be broken by the linter. The other strange part of the issue is, that it's only happening after my workflow is triggered by a push. Manually the workflow can be ran without the 'invalid yaml' issue happening. I am using a self-hosted gitlab, however it can also be reproduced on gitlab.com. (see example project)
Steps to reproduce
It can be reproduced using the following .gitlab-ci.yml file. The original variables and scripts have been removed from it because of data protection purposes, but it's still enough to reproduce the issue, I was able to do so in my project on a separate branch.
# and https://docs.gitlab.com/ee/ci/yaml/workflow.html#switch-between-branch-pipelines-and-merge-request-pipelines
# limitations
# * for include "`rules` keyword `changes` is not supported" :((
# https://docs.gitlab.com/ee/ci/yaml/includes.html#use-rules-with-include
# * for `changes` only branch pipelines or merge request pipelines are supported
# for `changes` on new branches (and in case of new tags) everything will seem changed (so changes will evaluate to true)
# https://docs.gitlab.com/ee/ci/jobs/job_control.html#jobs-or-pipelines-run-unexpectedly-when-using-changes
# TODO:
# - determine changes via git (so we need to run in a lightweight container where git is available)
# - solve caching for Rust (see cargo-chef and kaniko)
variables:
example: example2
workflow:
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
# - if: '$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS'
# when: never
- if: '$CI_COMMIT_BRANCH'
stages:
- build
- unit-test
- build-image
- e2e-test
- deploy
#no-changes:
# stage: build
# except:
# changes:
# - backend/**/*
# - frontend/**/*
# script:
# - echo "Nothing to build!"
# 888888888o .8. ,o888888o. 8 8888 ,88'
# 8888 `88. .888. 8888 `88. 8 8888 ,88'
# 8888 `88 :88888. ,8 8888 `8. 8 8888 ,88'
# 8888 ,88 . `88888. 88 8888 8 8888 ,88'
# 8888. ,88' .8. `88888. 88 8888 8 8888 ,88'
# 8888888888 .8`8. `88888. 88 8888 8 8888 88'
# 8888 `88. .8' `8. `88888. 88 8888 8 888888<
# 8888 88 .8' `8. `88888.`8 8888 .8' 8 8888 `Y8.
# 8888 ,88'.888888888. `88888. 8888 ,88' 8 8888 `Y8.
# 888888888P .8' `8. `88888. `8888888P' 8 8888 `Y8.
.backend:
rules:
- if: '$BE == "0"'
when: never
# - changes:
# - backend/**/*
# note, on new branches (and tags) this is always true (see limitations on the 4th line)
# cache:
# key:
# files:
# - backend/Cargo.lock
# paths:
# - backend/target/
# - .cargo/
backend test build:
extends: .backend
stage: build
image: rust
script:
- echo Test build
backend test:
extends: .backend
stage: unit-test
needs:
- backend test build
services:
- postgres:14-alpine
image: rust
script:
- echo Test
backend build container image:
extends: .backend
stage: build-image
needs:
- backend test
image: docker
services:
- name: docker:dind-rootless
command: ["--experimental"]
tags:
- dind-privileged
script:
- echo Building container image
.backend deploy:
extends: .backend
stage: deploy
needs:
- backend build container image
image: docker
environment:
name: invalid !!!
url: someurl
services:
- name: docker:dind-rootless
command: ["--experimental"]
tags:
- dind-privileged
script:
- echo Deploy
backend 1 DEV:
extends: .backend deploy
rules:
- if: '$BE == "0"'
when: never
- if: '$CI_COMMIT_BRANCH == "main"' # auto-deploy on main
# changes:
# - backend/**/*
- if: '$CI_COMMIT_BRANCH != "main"' # manual otherwise
# changes:
# - backend/**/*
when: manual
allow_failure: true
environment:
name: dev
backend 2 STAGING:
extends: .backend deploy
when: manual
allow_failure: true
environment:
name: staging
backend 3 PROD-TEST:
extends: .backend deploy
when: manual
allow_failure: true
environment:
name: prod-test
backend 4 PROD:
extends: .backend deploy
when: manual
allow_failure: true
environment:
name: production
# 8888888888 8 888888888o. ,o888888o. b. 8 8888888 8888888888
# 8888 8 8888 `88. . 8888 `88. 888o. 8 8 8888
# 8888 8 8888 `88 ,8 8888 `8b Y88888o. 8 8 8888
# 8888 8 8888 ,88 88 8888 `8b .`Y888888o. 8 8 8888
# 888888888888 8 8888. ,88' 88 8888 88 8o. `Y888888o. 8 8 8888
# 8888 8 888888888P' 88 8888 88 8`Y8o. `Y88888o8 8 8888
# 8888 8 8888`8b 88 8888 ,8P 8 `Y8o. `Y8888 8 8888
# 8888 8 8888 `8b. `8 8888 ,8P 8 `Y8o. `Y8 8 8888
# 8888 8 8888 `8b. ` 8888 ,88' 8 `Y8o.` 8 8888
# 8888 8 8888 `88. `8888888P' 8 `Yo 8 8888
.frontend:
rules:
- if: '$FE == "0"'
when: never
# - changes:
# - frontend/**/*
# note, on new branches (and tags) this is always true (see limitations on the 4th line)
frontend build:
extends: .frontend
stage: build
artifacts:
paths:
- frontend/dist/
image: node:18-alpine
cache:
key:
files:
- frontend/yarn.lock
paths:
- frontend/node_modules/
- frontend/.yarn
script:
- echo Build
frontend build container image:
extends: .frontend
stage: build-image
image: docker
services:
- name: docker:dind-rootless
script:
- echo Building container image
.frontend deploy:
extends: .frontend
stage: deploy
image: alpine:3.15
environment:
name: invalid !!!
url: someurl
needs:
- frontend build
script:
- echo Deploy
frontend 1 DEV:
extends: .frontend deploy
rules:
- if: '$FE == "0"'
when: never
- if: '$CI_COMMIT_BRANCH == "main"' # auto-deploy on main
# changes:
# - frontend/**/*
- if: '$CI_COMMIT_BRANCH != "main"' # manual otherwise
# changes:
# - frontend/**/*
when: manual
allow_failure: true
environment:
name: dev
frontend 2 STAGING:
extends: .frontend deploy
when: manual
allow_failure: true
environment:
name: staging
frontend 3 PROD-TEST:
extends: .frontend deploy
when: manual
allow_failure: true
environment:
name: prod-test
frontend 4 PROD:
extends: .frontend deploy
when: manual
allow_failure: true
environment:
name: production
# 8888888 8888888888 8 8888888888 d888888o. 8888888 8888888888
# 8 8888 8 8888 .`8888:' `88. 8 8888
# 8 8888 8 8888 8.`8888. Y8 8 8888
# 8 8888 8 8888 `8.`8888. 8 8888
# 8 8888 8 888888888888 `8.`8888. 8 8888
# 8 8888 8 8888 `8.`8888. 8 8888
# 8 8888 8 8888 `8.`8888. 8 8888
# 8 8888 8 8888 8b `8.`8888. 8 8888
# 8 8888 8 8888 `8b. ;8.`8888 8 8888
# 8 8888 8 888888888888 `Y8888P ,88P' 8 8888
deploy branch:
stage: e2e-test
needs:
- job: backend build container image
- job: frontend build container image
trigger:
project: myproject/k8s-resources
branch: main
strategy: depend
variables:
example: example2Example Project
https://gitlab.com/zozidalom/test
What is the current bug behavior?
Gitlab CI is unable to find the job in any previous stage, however the job is in one of the previous stages.
What is the expected correct behavior?
The pipeline should be executed with the jobs in the correct order.
Relevant logs and/or screenshots
Summary
Based on the original description and the examples in this issue, we can see that for the most part the error messaging is the issue. When a pipeline is created and some jobs are not created due to keywords like rules, needs or only, the error message is not clear as to why the job is missing, causing confusion.
The proposal to fix this issue is to improve the error messaging.
Proposed error message
"'#{name}' job needs '#{need[:name]}' job, but '#{need[:name]}' does not exist in the pipeline. This might be because of the only, except, or rules keywords. To need a job that sometimes does not exist in the pipeline, use needs:optional."
- Add a link from
needs:optionalto https://docs.gitlab.com/ee/ci/yaml/#needsoptional.
Below content is coming from an issue that was closed in favour of this issue.
If a pipeline has one or more jobs with needs keyword and that jobs cannot be run due to rules or any conditions, then a pipeline is not created at all and the error message is misleading
This is closely related to the issue: #224958 (closed), but, it only talks about schedules, and it ignores the fact that a pipeline should be created with rest of the jobs that satisfy the conditions.
Steps to reproduce
- Create a pipeline with following
.gitlab-ci.yml:
stages:
- build
- test
build-job:
stage: build
script:
- echo "build"
unit-test-job:
stage: test
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
when: always
script:
- echo "unit-test-job"
lint-test-job:
stage: test
needs:
- build-job
- unit-test-job
script:
- echo "lint-test-job"- This will not create a pipeline and shows
yaml invaliderror. Reason:'lint-test-job' job needs 'unit-test-job' job, but 'unit-test-job' is not in any previous stage. - Here, the
build-jobjob should have run as it does not have any rules. - Also, as we can now use
needsin the same stage, the error messageis not in any previous stagedoes not hold true in all cases.
Example Project
Project: https://gitlab.com/psureshbabu/no-pipeline-without-needs/ Pipeline: https://gitlab.com/psureshbabu/no-pipeline-without-needs/-/pipelines/909649498
What is the current bug behavior?
Error message is not clear, therefore it's unhelpful.
What is the expected correct behavior?
Provide a clear error message which indicates that a needed job was not created and point users to optional: need section in our documentation
Error/log entry here: is not in any previous stage
Relevant logs and/or screenshots
Output of checks
This bug happens on GitLab.com