Backend: Allow CI/CD mirroring to handle forks on GitHub SCM
Problem to Solve
When using CI/CD for External Projects, if the external project is forked, and a merge request is opened from the fork to upstream project, GitLab CI does not run CI pipelines. We should add support for forking workflows when using external projects with GitLab CI.
Intended users
Large projects (especially open source) accept pull requests from forks. Currently there is not way for them to run pipelines when a pull request is opened from a fork.
Current state
In https://gitlab.com/gitlab-org/gitlab-ce/issues/65139 we enable create pipelines for external pull requests when a GitHub repository is used with GitLab CI/CD. The merge request addresses pipelines for external pull requests coming from the same repository (no forks).
When GitLab CI/CD is set for a GitHub repository via the GitHub integration, a webhook subscription is created on the main repository in order to receive push
and pull_request
webhooks.
When a pull request is opened on GitHub, a pull_request
webhook is sent to GitLab. If the pull request is from the same repository, we save the pull request info in external_pull_requests
table and trigger a pipeline for external pull request if the commit SHA already is present in the repository mirror. If not, as soon as the mirror is updated (as consequence of the branch push
webhook), the open pull request is detected and a pipeline is created for it. When a pull request is updated on GitHub, a pull_request
webhook is sent to GitLab.
Proposal
- If a the pull request is from a fork repository, we still save the pull request info in
external_pull_requests
table - When a pull request is updated (
synchronize
action frompull_request
webhook) we fetchrefs/pull/{pull-request-iid}/head
and update mirror. This ensures that we have the code the pull request points to. Regardless whether it's from fork or same repository. This requires a Gitaly call to be implemented: gitaly#1913 - Finally create a pipeline for the external pull request ensuring the pipeline is set in
fork mode
(see Permissions and Security below)
The only difference between same repo PR and fork PR is the extra step where we fetch refs/pull/*/head
into the mirror repo so that the Runner can work off the pull request ref.
Permissions and Security
Security concerns for fork workflows are described here #11934 (comment 223099063) and regard both pipelines for merge requests and pipelines for external pull requests (GitLab CI/CD for GitHub repo).
The general plan, for the first iteration, is to limit the resources used by pipelines for forks by default (e.g. use only shared Runners, disable variables, disable access to clusters, etc.). Then we allow individual resources to be exposed by admins via project settings. Finally we should detect whether the pipeline triggerer is also a member of the parent project, with sufficient permissions to run a pipeline, and expose automatically resources the user should have access to.
Documentation
Testing
What does success look like, and how can we measure that?
- When a pull request is created from a fork repo on GitHub, GitLab should run a pipeline for the
refs/pull/N/head
and the SHA - When a pull request is updated from a fork repo on GitHub, GitLab should run a pipeline for the same
refs/pull/N/head
which should point to a new SHA - GitHub integration should still work as expected: e.g. GitLab should display status of the pipeline in the pull request page, block PR if pipeline failed, etc.
Links / references
- https://circleci.com/docs/1.0/fork-pr-builds/
- https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.