Create a persistent ref per pipeline and let runners fetch it
What does this MR do?
This MR creates a persistent ref refs/pipelines/:iid
per pipeline and let runners fetch it.
The overview of the process flow:
- The system ensures a corresponding pipeline ref exists when build status transits to running.
- Runners fetch code from the pipeline ref instead of source branch, refs/head or refs/merge.
- The system cleans up the pipeline ref when pipeline finished i.e. no more active jobs.
This fix effectively resolves the following problems
- When pipelines for merged results is enabled and source/target branch is advanced during MR pipelines are running, the pipelines fail because the target sha is not found in
merge_ref_path
. - When git-force-push is performed, it overwrites the target sha in the source ref, and result in the previous pipeline fails because source sha is not found.
Related: #14863 (closed)
Wed Sep 25 09:30:18 UTC 2019
Manual QA Case: Force-push to a branch
Scenario:
- A developer force-pushes to the same branch three times in a short interval (< 1 sec).
- Runner executes each pipeline and each job takes 5 seconds (sleep 5sec).
- Git strategy is clone, thus everytime runner clones a fresh repository (No cache as fetch)
Expectation:
- Runners successfully fetch/checkout source code from the given sha.
- Three pipelines have successful results.
- The persistent ref
refs/pipelines/:iid
have been cleaned up after pipeline finished (checked bygit ls-remote
)
Result: Pass
pipeline.ref
Case: Retry a job of a pipeline which ran on the non-existent sha on the associated Scenario:
- Same as above
- A developer retries a job of a non-latest pipeline.
Expectation:
- The persistent ref
refs/pipelines/:iid
is recreated before runner works on a job. - Runners successfully fetch/checkout source code from the given sha.
- The retried job have successful results.
- The persistent ref
refs/pipelines/:iid
have been cleaned up after pipeline finished (checked bygit ls-remote
)
Result: Pass
Case: Use pipelines for merged results and target branch is advanced
Scenario:
- "Merge pipelines will try to validate the post-merge result prior to merging" is enabled
- A developer creates a merge request and trigger a pipeline for merged result.
- The pipeline has three stages and each stage takes 5 seconds (Total 15 sec).
- The target branch is advanced during the pipeline execution (
git push origin target branch
)
Expectation:
- The pipeline for merged result successfully finishes. (This is the main problem to solve in #14863 (closed))
Result: Pass
Screenshots
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation created/updated or follow-up review issue created -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Performance and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. - [-] Tested in all supported browsers
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
- [-] Label as security and @ mention
@gitlab-com/gl-security/appsec
- [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
- [-] Security reports checked/validated by a reviewer from the AppSec team
Edited by Shinya Maeda