Working on some WIP MR's and was having multiple pipelines run when I was not ready to have pipelines start running, which seemed like a waste of resources to me.
This can also happen when more than one person are collaborating on an MR, and push up work on an MR to get some prelim feedback.
I believe this will save the company A LOT of resources.
How do I see this happening?
We can look at the title of the MR, maybe have something like NOPIPELINE like we have for WIP: on MR's.
We can also default it to not running the pipelines, and then someone needs to turn it on to have the pipeline run.
Pre push hooks for frontend linting
Create Merge Request/Branch in an issue will cause branch name to fail danger bot if issue title is long.
Awesome thanks for following up on this @jyavorska
I mean maybe we could start with quantifying how much a pipeline costs the company in terms of resources.
My above recommendation was just a brainstorm and POC. A potential easier way would be defaulting to not run the pipeline of Feature Branches and having a button that triggers the pipeline might be a little more simple.
Something that came up in %12.7 that might be related is if master is failing should new pipelines be kicked off until master has been resolved? (Just another use case we could potentially explore)
Thanks for the feedback. Maybe we can start outlining the times where you wouldnt want to run the CI.
Ie, clicking the create merge request button?
But i think thats a good start. My larger point is how to cut down on unnecessary pipelines running. This would save the company and the users money and pipeline resource.
Another example is i just pushed a commit and need to ammend my commit, when i force push now i have 2 pipelines running, but instead of EVERY TIME I PUSH, id like it to default to no pipeline being run.
@sstern naming your MR with SKIPCI in it would achieve the same thing, and still allow for situations where someone does want a pipeline run on MR creation (like if you create a branch with changes in it at the same time you make the MR).
One situation @sstern that results in a lot of pipelines is when an MR has 10 or so suggestions, and you click on "apply suggestion" on each one and it makes 10 pipelines. It would save a lot of resources if there was an "apply all suggestions" button and it did them all as one commit.
@jyavorska seems like a good opportunity to figure out how to batch apply suggestions via a queue that is built during the review process -- where all suggestions in the queue would be applied at the end of the review. If we went with an "apply all" button, we would need a way to discard suggestions beforehand -- which i don't think currently exists when reviewing an MR (or i just haven't seen it).
This would also probably alleviate the pain points around the "apply suggestion" button not reflecting the current state often times after it is pressed. I've found that I need to refresh an MR for it to tell me if it was successfully applied or not.
Scott Sternchanged title from Allow Disabling of Pipelines in MR's to save resources to Ideas for saving pipeline runs in MR's to save resources
changed title from Allow Disabling of Pipelines in MR's to save resources to Ideas for saving pipeline runs in MR's to save resources
Scott Sternchanged title from Ideas for saving pipeline runs in MR's to save resources to Ideas for saving pipeline runs in MR's to save resources from a frontend perspective.
changed title from Ideas for saving pipeline runs in MR's to save resources to Ideas for saving pipeline runs in MR's to save resources from a frontend perspective.
thanks, ive updated the description to identify some areas we can make some gains, we can promote this to an epic and start adding tickets. However you see fit.
thanks, ive updated the description to identify some areas we can make some gains, we can promote this to an epic and start adding tickets. However you see fit.
@sstern - Right now pipeline optimizations that reduce cost are tracked within &1853 but tracking cost savings issues separately would be more efficient. WDYT of promoting https://gitlab.com/gitlab-org/gitlab/-/issues/210601 to an epic and track related issues (like this one) with it.
@sstern - Another item to note is !27611 (merged). This will interrupt running pipelines that are created from frequent pushes or multiple suggestions being applied so the latest pipeline is the only running one.
Maybe a first iteration is just turning off the WIP pipelines then iteration No. 2 is applying the 10% of tests rule/test only changed files.
I think this is a good idea. If we can measure how many WIP pipelines are created on the GitLab project then we can prioritize this alongside other Efficiency items in flight for Engineering Productivity.
If WIP is stored as a true/false then we should do this. It's a two-way-door decision that can be undone easily if it disrupts Engineer's workflow.
I think people still rely on CI when in WIP but personally that’d be fine by me. We need to be cautious with breaking people’s workflow, though. We had the same problem when we wanted to move danger-review at the beginning of pipelines: people told us that they “don’t care” about danger-review while in WIP, but they want to see the test results.
There are legitimate use cases when you create a WIP MR to ensure it won’t be merged but definitely want pipelines to run for it (e.g. experimental changes). If we disable CI for WIP MRs, we would probably just end up with less WIP MRs (which is probably a good thing anyway), but not less pipelines.
I think that interruptible pipelines would have already saved us pipelines significantly. Disabling pipeline for WIP sounds like it would change workflow quite drastically. It would take manual effort for each MR author to remove WIP and then manually run pipeline. I’m not aware if there is a trigger to run a pipeline when WIP status is removed. Is there?
It seems this would be more disruptive than I had thought.
Can we confirm what users workflow is? Do we have any measurement on that? How many people actually do use WIP to check pipelines. In my personal flow I dont think ever used a WIP MR to check what happened on the pipeline. If I want to check specs or something wed run it locally (not saying my workflow is the only workflow).
Not sure if we can access that data, but if we can maybe we should measure and confirm before making that change?
Also, what are disrupted workflows worth? How much cost savings would we see if adding something like this, and how would it disrupt workflows and how crucial is it to the user.
In my personal flow I dont think ever used a WIP MR to check what happened on the pipeline. If I want to check specs or something wed run it locally (not saying my workflow is the only workflow).
Agreed, that was also my thinking when I created !27074 (closed): it'll be awesome for my workflow. But we need to acknowledge that "our" workflow isn't the only one.
Not sure if we can access that data, but if we can maybe we should measure and confirm before making that change?
How many people actually do use WIP to check pipelines
I do. I suspect others in Quality also do. The E2E pipelines take a long time and it's cumbersome to run all the different tests locally, so my workflow when working on E2E tests is to do as much as I can locally, then verify that all the tests pass in a WIP MR. Typically I then go through a few rounds of more changes as I discover that I've broken other tests.
I could do all that without using WIP, but it's a signal to others that I'm still working on it and so any unsolicited review should take that into account.
Instead of completely removing pipelines for WIP MR, we could also run a reduced set of stages, or make them manual. Perhaps review-app related jobs can be made manual during WIP. It is already only automated for frontend changes, but I'd imagine a WIP frontend change would get quicker feedback in a local environment.
Having said that, my bigger concern is that there might be possibility of an MR getting merged without going through the full pipeline. For example, a reduced pipeline is created during WIP stage, it passes. As soon as the MR WIP status is removed, the MR can be merged.
How many people actually do use WIP to check pipelines.
I always do this, and I also believe a lot of people are doing this. I keep WIP before passing to a reviewer, and I mark WIP again when I think I need to make some significant changes to address the reviewer's feedback.
Basically, WIP is for me is a signal for reviewers that I am ready for another round of review.
If we don't run pipelines with WIP merge requests, that means I'll:
Always run tests locally. This is not always feasible
Always keep the merge requests not in WIP
This is actually defeating the purpose of WIP to me. So I second to @mlapierre here that:
I could do all that without using WIP, but it's a signal to others that I'm still working on it and so any unsolicited review should take that into account.
In the end I can just stop using WIP, but I don't think that's solving the issue we're aiming here.
I suggest we look into #27955 if we want to skip pipelines intentionally.
Scott Sternchanged the descriptionCompare with previous version
changed the description
Scott Sternchanged title from Ideas for saving pipeline runs in MR's to save resources from a frontend perspective. to Ideas for saving pipeline runs in MR's to save resources.
changed title from Ideas for saving pipeline runs in MR's to save resources from a frontend perspective. to Ideas for saving pipeline runs in MR's to save resources.
Scott Sternchanged the descriptionCompare with previous version
Can be a side initiative that we can work together. I have definitely been in a place where I had to rebase multiple times because I violated this rule when working on documentation MRs. I think we should allow more char limit.
Can be a side initiative that we can work together. I have definitely been in a place where I had to rebase multiple times because I violated this rule when working on documentation MRs. I think we should allow more char limit.
As explained in #119340 (comment 268045719), there shouldn't be any need of rebasing/amending the commits since the squash option should be enabled for MRs that have suggestions applied.
@meks as this is currently assigned to you, is the labelling of this issue correct with ~"group::continuous integration" applied? I don't believe our engineers in CI are currently working on it and it is scheduled in the current %13.1 milestone. Thanks!
@cheryl.li I believe a number of improvements linked in the issue description has already been completed by the Engineering Productivity team. https://gitlab.com/gitlab-org/gitlab/-/issues/210601. I believe the label ~"group::continuous integration" was applied since the work here relates to our CI configuration and we want to keep collaboration and awareness open.
@sstern@thaoyeager what are your thoughts if we are to close this issue and track the other improvements in the remaining issues?
I think another possibility that we discussed was to not run pipelines if the MR has WIP: I am sure there is another issue tracking this somewhere along with some of the challenges that we have uncovered.