Backend: Allow CI/CD mirroring to handle forks on GitHub SCM

Problem to Solve

When using CI/CD for External Projects, if the external project is forked, and a merge request is opened from the fork to upstream project, GitLab CI does not run CI pipelines. We should add support for forking workflows when using external projects with GitLab CI.

Intended users

Large projects (especially open source) accept pull requests from forks. Currently there is not way for them to run pipelines when a pull request is opened from a fork.

Current state

In https://gitlab.com/gitlab-org/gitlab-ce/issues/65139 we enable create pipelines for external pull requests when a GitHub repository is used with GitLab CI/CD. The merge request addresses pipelines for external pull requests coming from the same repository (no forks).

When GitLab CI/CD is set for a GitHub repository via the GitHub integration, a webhook subscription is created on the main repository in order to receive push and pull_request webhooks.

When a pull request is opened on GitHub, a pull_request webhook is sent to GitLab. If the pull request is from the same repository, we save the pull request info in external_pull_requests table and trigger a pipeline for external pull request if the commit SHA already is present in the repository mirror. If not, as soon as the mirror is updated (as consequence of the branch push webhook), the open pull request is detected and a pipeline is created for it. When a pull request is updated on GitHub, a pull_request webhook is sent to GitLab.

Proposal

• If a the pull request is from a fork repository, we still save the pull request info in external_pull_requests table
• When a pull request is updated (synchronize action from pull_request webhook) we fetch refs/pull/{pull-request-iid}/head and update mirror. This ensures that we have the code the pull request points to. Regardless whether it's from fork or same repository. This requires a Gitaly call to be implemented: gitaly#1913
• Finally create a pipeline for the external pull request ensuring the pipeline is set in fork mode (see Permissions and Security below)

The only difference between same repo PR and fork PR is the extra step where we fetch refs/pull/*/head into the mirror repo so that the Runner can work off the pull request ref.

Permissions and Security

Security concerns for fork workflows are described here #11934 (comment 223099063) and regard both pipelines for merge requests and pipelines for external pull requests (GitLab CI/CD for GitHub repo).

The general plan, for the first iteration, is to limit the resources used by pipelines for forks by default (e.g. use only shared Runners, disable variables, disable access to clusters, etc.). Then we allow individual resources to be exposed by admins via project settings. Finally we should detect whether the pipeline triggerer is also a member of the parent project, with sufficient permissions to run a pipeline, and expose automatically resources the user should have access to.

Documentation

Testing

What does success look like, and how can we measure that?

• When a pull request is created from a fork repo on GitHub, GitLab should run a pipeline for the refs/pull/N/head and the SHA
• When a pull request is updated from a fork repo on GitHub, GitLab should run a pipeline for the same refs/pull/N/head which should point to a new SHA
• GitHub integration should still work as expected: e.g. GitLab should display status of the pipeline in the pull request page, block PR if pipeline failed, etc.

Links / references

• https://circleci.com/docs/1.0/fork-pr-builds/
• https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.