Skip to content

GitLab Next

Merged
Created by Shinya Maeda@shinya.maeda💡Maintainer8 of 8 tasks completed8/8 tasks

Create a persistent ref per pipeline and let runners fetch it

What does this MR do?

This MR creates a persistent ref refs/pipelines/:iid per pipeline and let runners fetch it.

The overview of the process flow:

  • The system ensures a corresponding pipeline ref exists when build status transits to running.
  • Runners fetch code from the pipeline ref instead of source branch, refs/head or refs/merge.
  • The system cleans up the pipeline ref when pipeline finished i.e. no more active jobs.

This fix effectively resolves the following problems

  • When pipelines for merged results is enabled and source/target branch is advanced during MR pipelines are running, the pipelines fail because the target sha is not found in merge_ref_path.
  • When git-force-push is performed, it overwrites the target sha in the source ref, and result in the previous pipeline fails because source sha is not found.

Related: #14863 (closed)

Manual QA Wed Sep 25 09:30:18 UTC 2019

Case: Force-push to a branch

Scenario:

  • A developer force-pushes to the same branch three times in a short interval (< 1 sec).
  • Runner executes each pipeline and each job takes 5 seconds (sleep 5sec).
  • Git strategy is clone, thus everytime runner clones a fresh repository (No cache as fetch)

Expectation:

  • Runners successfully fetch/checkout source code from the given sha.
  • Three pipelines have successful results.
  • The persistent ref refs/pipelines/:iid have been cleaned up after pipeline finished (checked by git ls-remote)

Result: Pass

Case: Retry a job of a pipeline which ran on the non-existent sha on the associated pipeline.ref

Scenario:

  • Same as above
  • A developer retries a job of a non-latest pipeline.

Expectation:

  • The persistent ref refs/pipelines/:iid is recreated before runner works on a job.
  • Runners successfully fetch/checkout source code from the given sha.
  • The retried job have successful results.
  • The persistent ref refs/pipelines/:iid have been cleaned up after pipeline finished (checked by git ls-remote)

Result: Pass

Case: Use pipelines for merged results and target branch is advanced

Scenario:

  • "Merge pipelines will try to validate the post-merge result prior to merging" is enabled
  • A developer creates a merge request and trigger a pipeline for merged result.
  • The pipeline has three stages and each stage takes 5 seconds (Total 15 sec).
  • The target branch is advanced during the pipeline execution (git push origin target branch)

Expectation:

  • The pipeline for merged result successfully finishes. (This is the main problem to solve in #14863 (closed))

Result: Pass

Screenshots

Does this MR meet the acceptance criteria?

Conformity

Performance and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team
Edited by Shinya Maeda