CI linter: Extend errors messaging to include warnings

Problem to solve

As we start deprecating CI features like extending only/except in shared templates gitlab-org/release/tasks#1315 (closed) it would be helpful to have a linter that helps users update their CI before it breaks. In order for this to be the most useful it should run on every pipeline and display them somewhere prominently.

Expand to an example

Travis does this nicely, showing linting panel for every job, for example:

Screen_Shot_2020-05-04_at_8.57.05_AM

Intended users

Personas are described at https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/

User experience goal

Proposal

For this MVC:

  1. Using rules without workflow: rules; link to the workflow:rules docs
  2. Using rules and only/except in the same job is not supported
(expand here for part of original proposal that is scoped to issue #219431 (closed))

Store each offenses, one per line, as part of YAML errors, each identified with a severity, such as:

  • error:
  • warning:
  • deprecation:

The format of a single line message could be:

<severity>: <job-name>: <message>

We could use this yaml_errors to also further refine the Build View and show this information there

Today, we would do these things:

  • we just refine this box to present it differently on frontend based on severity
  • we would need to add a small change to Ci::Config to allow easy way to generate warning and deprecations, join them together and persist as part of ci_pipelines.yaml_errors, including a context in above format

Execution plan

  1. Replace has_yaml_errors? with config_error? so that it won't use yaml_errors which will hold a different type of messages with this feature
  2. Add mechanism inside YamlProcessor that allows to append warning messages as we do today with errors. Warning messages could be used to also describe deprecation warnings. We could distinguish them later.
    • Maybe we could extend Gitlab::Config::Entry::Configurable to have helper methods such as add_warning
  3. Store multiple messages in Ci::Pipeline#yaml_errors. Ensure that pipeline.errors don't include warnings or deprecations.
  4. Expose warnings in the frontend and API response for the Linter / Pipeline view - this marks the 1st iteration, from user perspective

In preparation to work on this maybe we need to first open a MR about replacing the use of has_yaml_errors? with config_error?. Then, we can make yaml_errors orthogonal to failure_reason as we may not have any failure reasons but warnings in yaml_errors.

Further details

Permissions and Security

Documentation

Availability & Testing

  • Unit test changes - Yes
    • Ensure renaming does not affect existing behaviors - Error messages should still be displayed
    • Warnings message should show up where there is violation even if pipeline did not fail
  • Integration test changes - Yes
    • Integration test API ~> UI:
      • Warnings and deprecation messages are recognized and displayed accordingly
      • In case pipeline did fail with a config error and there are also warning violation, show both error and warning in the banner.
  • End-to-end test change - not required

See the test engineering planning process and reach out to your counterpart Software Engineer in Test for assistance: https://about.gitlab.com/handbook/engineering/quality/test-engineering/#test-planning

What does success look like, and how can we measure that?

What is the type of buyer?

Is this a cross-stage feature?

Links / references

Edited by Thao Yeager