Optimize subsequent pipeline executions based on both last execution status and last pushed files
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Problem to solve
Internally at GitLab, MRs that change files in different parts of the application will result in triggering a pipeline with lots of stages and jobs based on the files diff, and this increases the feedback loop, but once the pipeline is green, and code review starts, suggestions in a smaller portion of code might arise, and they may need to be addressed before the MR is merged. In these cases, instead of running a subset of the pipeline, based only on the latest changes (files), we re-run the whole pipeline, and in worst cases, unrelated jobs that have passed previously ought to fail due to some kind of instability, which in the end increases a lot the feedback loop unnecessarily.
Intended users
Engineers at GitLab
Further details
Having a mechanism that would trigger a subset of the last green pipeline based on the files changed in the last push would:
- Reduce the feedback loop (quicker feedback)
- Reduce infrastructure costs (less runners needed in some pipeline runs)
- Allow us to release features more frequently and move faster
Proposal
It would be awesome if GitLab CI could have a mechanism to trigger a subset of the pipeline depending on:
- The last full pipeline execution status
- The files that changed between the last green pipeline execution and the last push
Permissions and Security
TBD
Documentation
TBD
Testing
TBD
What does success look like, and how can we measure that?
Success metric
- Reduction on time from opened to merged MR
Acceptance criteria
Let's imagine the following use case.
An MR is created to replace an end-to-end test that is flaky, with a lower-level test, that besides running faster, is also more reliable. The MR changes the following files: An end-to-end test file is deleted (the one that defined the flaky test), a new Rspec test is created (the one that implements the deleted test in a lower-level), a view file (*.html.haml or *.vue), to add a data attribute in an HTML element for testability purposes; and a page object file (to delete element class attributes and methods that were only used by the deleted test).
Everything was implemented with care, and so, in the first pipeline execution, all stages are green. Let's imagine that the following stages were run: stage 1 - static analysis (~ 1 minute to run); stage 2 - Rspec tests (~ 5 minutes to run); and stage 3 - end-to-end tests (~ 10 minutes to run).
Feedback time from the pipeline: ~ 16 minutes.
The Code review process starts.
The changes look good, but there could be a small improvement in the Rspec test specifically, which would make the test expectation better scoped, which helps in debugging the test in case it fails in the assertion step. This change will have no impact in the deleted test, nor in the page object or the application itself. It will only impact the Rspec file. But, because the pipeline is configured to generate the stages based on the full files diff of the whole MR, when this new (small) change is committed and pushed, the same pipeline will run again, and it will take approximately 16 minutes again to give us feedback about the last change.
If we had the suggested mechanism of a new pipeline being generated and triggered based on the last full pipeline status, and the last pushed files, because the only changed file was the Rspec test, a new pipeline would be generated and triggered, and it would be composed by the following stages (stage 1 - static analysis (still ~ 1 minute to run); and stage 2 - Rspec test (~ 2 minutes to run). There would be no stage 3 because we know that they were green in the last execution, and the last push touched a file that has no impact in nothing related to end-to-end tests, nor the application itself, it's just a Rspec file, and so, the third stage would be a waste time.
Notice also that stage 2 (Rspec test) executed faster because instead of re-running the full RSpec suite, only the changed file was executed since these kinds of files are completely independent of each other. In the end, the new pipeline would give use feedback in approximately 3 minutes, which would save us 13 minutes.
Feedback time from the new pipeline: ~ 3 minutes.
Time saved: ~ 13 minutes.
With that, the acceptance criteria could be a "before-and-after" comparison from time to create and have an MR merged, based on the above example, and the time should be shorter.
What is the type of buyer?
¯_(ツ)_/¯
Links / references
Cc @jlenny