Downstream QA tests fail for MR from forked projects
Summary
Contributors working on forks of secure projects see failures on downstream qa jobs.
See this MR and its pipeline for an example.
Here's what the same code looks like when copied and triggered in the project: the MR.
This behaviour occurs because forks are not permitted to run downstream pipelines. See this issue for more context.
The root cause and solutions are discussed here.
Steps to reproduce
- Fork an analyzer project (i.e.
gemnasium-maven
) to personal account. - Make some changes and create an MR.
- Wait for pipeline to get to the downstream jobs. They will fail.
Example Project
Example MR from a gemnasium-maven
fork: gitlab-org/security-products/analyzers/gemnasium-maven!21 (merged)
What is the current bug behavior?
Downstream qa jobs fail without a real indicator as to why this occurred.
What is the expected correct behavior?
Ideally, the downstream qa jobs should succeed, but this is currently in discussion due to security and permission concerns (#11934 (closed)).
At the very least, instructions need to be given to the user as to why the downstream QA jobs are expected to fail.
Relevant logs and/or screenshots
See the pipeline created by an MR in the fork: https://gitlab.com/fcbrooks/gemnasium-maven/pipelines/102278884 See the pipeline created by an MR in the analyzer project: https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/pipelines/103840481
Possible fixes
This has been discussed in the following open issue as a feature to allow the behaviour: #11934 (closed)
What the secure team can do in the meantime is document this behaviour for contributors.
Additionally, we may consider allowing failure on the downstream pipelines so that the MRs can succeed and be merged after manual testing.
The testing solution for MRs from forks seems to be manual testing. There are several options as to how to allow this:
- Instruct contributor to add tests for the relevant project manually (push a branch to the qa/test project pointing to their MR docker tag for testing). This is pretty complex and tedious, but it might be made easier by creating a template for creating these.
- Have the approver in secure copy the MR, and create their own MR in the analyzer, thus triggering the downstream tests. The downside is that this is very time-consuming and error-prone.