Backend: needs: optional: true with include: rules is causing pipeline errors

Existing behavior

Today it's impossible to use needs: to create a job dependency that points to a job added with include:local:rules. When the configuration is checked for validity, GitLab returns undefined need: <job-name>. this issue is to improve this behavior.

Summary

When the needs: optional keywords are used with include: rules, the pipeline fails with the message "child-job-name job: undefined need: parent-job-name". According to our docs, the needs: optional keyword can be used to "need a job that sometimes does not exist in the pipeline". In this case, if we use the include: rules keywords to control when a file with jobs in it gets included, the pipeline fails.

Steps to reproduce

• Create a new project.
• Add two yml files:

.gitlab-ci.yml

include:
  - local: .hello-world.yml
    rules:
      - if: $RUN_HELLO_WORLD == "true"

depends-on-hello-world:
  needs: 
    - job: hello-world
      optional: true
  script:
    - echo "This job depends on Hello World!"

.hello-world.yml

hello-world:
  script:
    - echo "Hello World!"

1. Attempt to run the pipeline by sending RUN_HELLO_WORLD=true, the pipeline runs.
2. Attempt to run the pipeline by sending RUN_HELLO_WORLD=false, the pipeline fails.

Example Project

Example Project: https://gitlab.com/b_freitas/needs-optional-and-include-with-rules

• Branch that reproduces the bug, using needs:optional with include:rules:if
• Branch that works, using needs:optional with include only (without rules:if under include)

What is the current bug behavior?

The pipeline does not start and we get the following error message: "depends-on-hello-world job: undefined need: hello-world".

What is the expected correct behavior?

If the RUN_HELLO_WORLD variable is set to true the pipeline should run with the two jobs. If the RUN_HELLO_WORLD variable is set to false the pipeline should run with one job (depends-on-hello-world) and ignore the other one (hello-world).

Relevant logs and/or screenshots

Output of checks

This bug happens on GitLab.com

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Possible fixes

Steps to include files and run the pipeline;

• include the YAML files (we can't include .hello-world.yml because of the rule)
• validate all needs independently from optional
• validate all needs with optional

I guess we may fix this by using the optional attribute in the 2nd step. So, probably we need to adjust Gitlab::Ci::YamlProcessor to use optional.

Implementation

backend	MR
Fix YamlProcessor to allow needing an optional, undefined job	!116335 (merged)