Release evidence collection results in test results from inconsistent pipelines

Summary

Release evidence collection of test results, as implemented in #32773 (closed), results in test results collected from different pipelines depending on release creation mechanism and timing, and in some cases from incomplete pipelines.

Steps to reproduce

Create a release using the web UI or API as described below. The examples below are do not include future releases.

Example Project

An example project is at https://gitlab.com/aarongoldenthal/release-evidence-test, with the various cases outlined below.

What is the current bug behavior?

If a tag and release are created together via the GitLab UI, then the release evidence is taken from the last pipeline (in the example project above this was a push pipeline), as expected.

If a tag is added to an existing commit (whose last push pipeline is complete), and in that pipeline the release is created via the API, the release evidence is taken from the tag pipeline. The evidence is pulled at the point of release creation, even if that pipeline is not complete, and it pulls only the evidence available at that point (e.g. if the release is created after the test stage then it pulls test results from that pipeline, but if the release is created before the test stage or with no test stage then no results are returned).

In addition to this creating at least non-intuitive, and arguably inconsistent results (since API and UI created releases result in different evidence), the built-in security jobs are not run in tag pipelines. So, they are included in release evidence created from web UI, but not the API.

What is the expected correct behavior?

Releases should:

  • Produce consistent release evidence when created via any available means
  • Allow creation from a tag pipeline and include the appropriate release evidence without having to repeat the tests in that pipeline
  • Pull release evidence from the last push pipeline for a branch that includes the tag, and wait until that pipeline completes if it is not when the release is created. This allows inclusion of the predefined security and other template jobs, and other test jobs, and only run in pipelines where content is changed.
    • The release evidence could be pulled from the tag pipeline once complete, but in that case the predefined security and other template jobs need to run in tag pipelines as well (which can add a lot of compute time).

Output of checks

This bug happens on GitLab.com.