Merge train/Release train/Merge when master succeeds: run build on merged code before merging

@fzahle It would be a nice feature, but this would also have a big impact on performance. Since we want to make sure that tests will pass after merging changes, we would need to trigger build for every commit that drops into master after merge request has been created. And while I've seen projects with a few hundreds of contributions to master each day, this can be quite painful. What do you think ?

cc @ayufan

Added ~19173 label

I see that it could be an issue for large public projects as you describe, but wouldn't you always have to test a feature branch up against the latest master to ensure that a merge into master would not break anything?

A way to get the desired workflow with the current version of gitlab would be that once the feature branch is ready, the devs push to a branch on the main project that merges the MR and master, and name the branch in a certain way so the CI picks it up e.g. MR:feature-XX. A merge request is then made and if this passes the owner can safely merge it into master. But it would just be nice to have all this automated in the merge request workflow.

So it would make sense to make this an option in project settings, so depending on the type of workflow used in the project, project owners can turn this feature on.

@fzahle I would be a nice feature to have, for sure. As far as I know it is/was considered to introduce such workflow into GitLab, but this is also not an easy thing to do.

Not saying that we will not see something like that in GitLab. What do you think @JobV ?

Just for future reference here's some "duplicate" issues: #3770 (closed) and #3581 (closed).

I tend to agree with @fzahle, this should at the very least be an option in project settings. At my work, we've resorted to manually merging the source branch with master in our build script. Unfortunately, the destination branch isn't provided to the runner, so this can fail if someone is merging into any other branch.

Title changed from Trigger CI build from merge request to Run build on merged code before merging

Added ~14213 label

mentioned in issue #3770 (closed)

mentioned in issue #3581 (closed)

We want to do this in the future, but it's not arbitrary. I'll put it for 8.7 for now.

Milestone changed to 8.7

awesome, thanks!

Awesome! Thanks @JobV

mentioned in issue #4710 (closed)

Also requested on https://twitter.com/greg_diamond29/status/669989246053273600
http://homu.io/ already does something like this for Travis CI.
This should be an optional feature.
If this is not contributed we might opt to make it a GitLab EE feature.
Now that we have Auto-merge on Build Success we can reuse that.
A future feature should allow you to make a ci-train (I click auto-merge on build success on branch B, branch A is still testing, branch B will test with the merge of A in master and then B in master). :train_diesel:

A few different modes are possible:

Retest all merge requests when the target branch changes (master), need lots of runners.
Test a merge request once when new commits are pushed, don't update when master changes, test one more time directly before merging.
Only test just before merging (saves at least 50% compared to the previous option).

What mode do people prefer in the first iteration?

Added ~123454 label

as a first iteration you could simply add a Build button to the merge request view, so the devs manually decide when to build. It could be that some commits that come into master are cosmetic and others require running the test suite to make sure it works. Only privileges >= developer, master, owner should be able to fire off the build.

I of course vote for this to go in the CE in some shape or form, since it seems a widely wished for capability. The more fine grained modes you list above could then go in the EE.

@fzahle I love how we're trying to simplify it. So you propose a 'Build merge with target branch' button in the merge request? Should that be 'Automerge on merge with target branch success' so we save people work? As for CE/EE, if people contribute it CE, otherwise EE since it is more useful for larger teams.

@sytses I would make the 'Automerge on merge with target branch success' optional. Proposed code changes could cause unwanted behaviour such as increase in computational time that the test suite may not be able to capture.

@fzahle OK, I understand, this does however complicate the MR interface that is already pretty hard. Why can't the test set fail if the computational time is exceeded?

@sytses it was just an example, I just figured that people probably have different workflows, so making the first iteration as manual as possible would make it more agreeable by everyone. I don't think the merge request view is particularly cluttered. The box with the build and merge status could simply have a button with "Run Build" and some text indicating if the build has run on the current head of the target branch.

@fzahle then it needs two buttons (merge and build), with automerge it would have one (merge if build succeeds), currently there is one button on all MR's and conditional UI changes are hard.

@sytses ok then. Your proposal is not a deal breaker at all for me, so I'm perfectly fine with that if my proposal is complicated to implement.

OK, so first iteration is:

A project setting that says: 'before merging test the merge with target'.
All commits will still be tested as normal (mode 2).
Every merge request has a button that says 'Merge if merge result is green'
If that button is pressed it should only merge if target is still at the position it tested.

great

Title changed from Run build on merged code before merging to Run build on merged code before merging (EE?)

I think it should merging with the target branch instead of master. In my company we always merge to a develop branch. So using target branch would be much more helpful than master.

Added ~122770 label

@bdandoy I agree, I changed https://gitlab.com/gitlab-org/gitlab-ce/issues/4176#note_3065776

If that button is pressed it should only merge if target is still at the position it tested.

This makes sense. For a project like gitlab-ce, where there are commits often in the time that any builds runs (which takes a long time), this would mean this would not work.

@JobV If you mean commits to master, indeed it would be a problem. In that case you need the ci-train function I talked about before.

I've used Homu a lot, and I really enjoy the workflow they have, which will automatically try to build + merge once seeing an approval. I'd be delighted to see a similar flow come to GitLab, so I'd like to propose a companion feature.

On this issue of running builds on merged code, I think I agree with the proposal by @sytses, but just to be sure, here's what I would like to see:

A project setting that says: 'Before merging a MR, run builds on a merge with the target' or similar.
All commits will still be tested as they are now: builds should run on each commit itself, not on a merge commit.
When the merge button is hit, then a merge commit will be created to merge the branch into the target, and builds will run on the merge commit. If the builds succeed, then fast-forward the target branch into the merge commit, merging the MR and closing it successfully. If any builds fail, then abort the merge and report the build failures back to the MR.

(This seems to be mode 2 or 3, but I'm not sure.) This means that each commit that ends up in the final git history gets builds run on it, and runs the fewest necessary builds to achieve that property. As for the ci-train function, bors and homu have a similar feature to handle high-velocity projects, which they call rollups, and it seems to work pretty well, so +1 for that.

The companion feature would be an option to automatically merge once the approval threshold is reached. This option would only be available if approvals are enabled. Essentially, the approve and merge button get merged: once a merge request reaches the approval threshold, it will automatically (try to) merge.

I think both of these features may be valuable on their own, and even more valuable together, allowing for one-click safe merges. What do you think of this approach?

@aneeshusa Your proposal is flawed. Lets use Gitlab-CE as an example:

Here's the last set of builds for master. With your proposal, there's a 32 minute delay between clicking merge and it actually merging. That's assuming the merged build actually succeeds, merges should happen instantly.
In the event that it fails:
DevA pushes their branch and creates a MR
Gitlab starts building the branch
DevB, DevC and DevD review these changes
They approve the changes
The build succeeds after 32 minutes of running
The branches are merged and a build starts
32 minutes later the build fails and the merge is rejected
DevA fixes their branch and pushes the changes
Gitlab starts building the branch, again
DevB, DevC and DevD review the changes, again (if DevA amends their commit, they will have to review all the changes again)
They approve the changes, again
The branches are merged and a build starts, again
32 minutes later the build succeeds and the merge occurs. After 96 minutes DevA has their changes merged in.

Alternatively, had Gitlab built the merged branch instead, DevA would have been aware after 32 minutes that they needed to merge/rebase their branch. I'd rather know my branch needs to be fixed straight away, than after everyone has reviewed it and it has been built twice. Also, had Gitlab built the merged branch instead, it would have built fewer times than your proposal, which you've claimed runs the fewest necessary builds.

I'll concede that their might be a case where the merged result will build but the source branch won't, which would result in a commit in master being broken. But unless every commit in a branch is tested, there's no guarantee that this won't occur normally. Not to mention that the likelihood of this occurring is not worth the huge delays in a MR going through.

And to top it all off, the destination branch would need to be built after being merged. Otherwise, build scripts will have a random merged branch name instead of "master", for example. How do you configure a deploy script to only run on the master branch when it never actually gets built? Do you increase the complexity of the YAML parser, or where ever the ref list is tested, to handle this merged weirdness?

On HN https://news.ycombinator.com/item?id=11092756 someone mentioned that Travis CI runs the build on the current merge. This makes a lot of sense to me and seems like the simplest way to start. "Rather than test the commits from the branches the pull request is sent from, we test the merge between the origin and the upstream branch." https://docs.travis-ci.com/user/pull-requests

Alternatively we can have a 'merge if build of merged result succeeds button". But in case master moves on before the build is ready the MR will have to be retested.

Title changed from Run build on merged code before merging (EE?) to Run build on merged code before merging (EE option?)

https://gitlab.com/gitlab-org/gitlab-ce/issues/15122 is also going in the direction of 'merge if the build succeeds'.

This would evolve the pipeline https://gitlab.com/gitlab-org/gitlab-ce/issues/3743 into a 'release train' where build are tested and committed in sequence.

mentioned in issue #15122 (closed)

Thanks @sytses for identifying my issue (#15122 (closed)) as a duplicate of this. I would recommend those considering how to implement this feature check out my issue, as I spent a lot of time trying to explain exactly what's happening with GitHub/Travis, and how GitLab CI could do it even better.

Also, my issue points how this would be a relatively minor change to the existing GitLab CI functionality, and I personally, don't see it warranting an EE-only sticker.

Furthermore, I've started taking notes on the technical implementation at https://gitlab.com/jonathon-reinhart/gitlab-ce/issues/1.

Milestone changed to 8.9

@jonathon-reinhart great issue and analysis.

This issue is indeed a serious drawback of a GitLab CI that:

you can't configure a runner on the main repository and make it automatically test all merge requests from forks (after the main repository's admin approving each fork, obviously) without the forks having to setup their own runner --- which is a pain (more on this below), and then all the local branches on forks get tested, which we don't want. We only want to test merge requests submitted against the main repository.
you can't easily test the merge commit --- though the workaround here is to simply tell the merge request author to merge with master, so this by itself is not that big of a deal.

In order to enable a specific runner for forks, the only way is to create a dummy repository and enable the specific runner there. Then you add people that you trust as developers into this repository. Then these people will see your specific runner as an option in the settings of any of their repositories, and so they can enable your specific runner (e.g. which can have some special setup for HPC clusters) on their forks of your main repository. (Otherwise they can't see your specific runner and can't enable it on their fork, and thus a merge request will not be tested.)

Solution

Merge request triggers a test on the branch and the merge commit.
The merge commit test is run using the target repository's runner (after the main repository's admin approves each fork --- if a fork is not approved, then no test is run for security reasons).
The branch test is run using the source (fork) repository runner (or none if none is configured in the fork). If the source repository is the same, i.e. the merge request is created from a local branch, then you get two tests --- one for the local branch, one for the merge commit (that's how Travis-CI works).
This should be configurable, so you could tell it to only test the merge commit, or only the branch.
You should also be able to retest a merge request, which would test the latest merge commit (if master got updated in the meantime), optionally even doing this automatically (on github you have to close/reopen a pull request to trigger a new build, but Jenkins can be configured to retest each merge request automatically whenever the target branch (master) gets updated).

@certik Thanks. Much of what you've said is covered by proposal. Recall that the source branch for a merge request (even from a fork!) is already fetched into the target repository, under refs/merge-requests/<number>/head. So in this case, there's really nothing too special about an MR from a fork.

I agree, there could be more done with allowing forks to use the original project's CI for local development branches.

Also, I disagree that the merge request should trigger a test on the branch and the merge commit:

The MR approver doesn't really care if the code built by itself; he only cares what merging the branch will do to the project he is responsible for.
A build for the branch itself was already done on the push that had to happen before the MR could be opened.

mentioned in issue #3743 (closed)

@ayufan @markpundsack this will be up to you to schedule / prioritise or at least something we should collaborate on.

Milestone changed to Backlog

started looking at gitlab ci runners today with the assumption this was already possible. Kinda disappointed it's not and now i'm back to our old solution (a somewhat less than ideal integration with jenkins given that we don't have EE yet). Would like to see this feature added.

Thanks!

mentioned in issue #19091 (closed)

@sytses asked me to provide some input from our perspective, so:

Servo and Rust use Homu on GitHub. One of the core ideas here is that master must always pass, so instead of merging things whenever they are approved and tests pass, approvals put things in a queue, and the bot tests the top PR against master, merges if it passes, and continues down the queue in this fashion.

Here are some examples of queues: Rust, Servo. We can prioritize items in the queue, and do things like sending to "try" (run full testsuite, but don't merge -- this is useful because both projects use buildbot for the merge-testing and have travis just for humans to check things; though travis integrates with homu too). There is only one PR running at a time, not counting try PRs.

We may add gitlab integration in the future -- the project has always been undermaintained but there are recent efforts from many different projects using this to jointly maintain it, so this could happen

Anyway, this tool might provide some insight on the kind of approve-test-merge workflow that others find useful. Probably shouldn't be the default though. Let me know if you have questions!

@Manishearth I think you lost part of your comment after:

so instead of merging things whenever they are approved and tests pass, approvals put things in a queue, and the (...?)

Thanks for the comment @Manishearth

@agilob ah, sorry, fixed :)

mentioned in issue #19627 (moved)

mentioned in issue #20043 (closed)

@gitlab team: Will this be added? Any ETA? few months now and no updates

I haven't seen Jenkins mentioned here, but can be used to build a merge request or feature branch and push to the target branch only on success. That is another alternative/competitor to GitLab CI in this area. On that note, I have personally developed some extensive custom GitLab/Jenkins integration that is used by dozens of projects at my company. While it's more work than using the built-in GitLab CI, it is much more flexible.

just for everyone's benefit we added the following to our .gitlab-ci.yml file to accomplish this and it has been working well for us:

before_script:
  - git merge origin/master --no-commit --no-ff

The one downside is if you are working on a change that won't be going to the master branch the results can be misleading. The only option to rectify this at the moment would be to (within the gitlab-ci.yml file):

query the gitlab API and determine if the commit is associated with a merge request
use the target branch of the merge request
otherwise default to master

OR

Change your gitlab-ci.yml file for commits that will be going to a different branch (kinda seems silly).

What would be awesome is to add few more environment variables while triggering builds from MRs something like the hooks: ref: http://docs.gitlab.com/ce/ci/variables/README.html

CI_MERGE_REQUEST_SOURCE_BRANCH
CI_MERGE_REQUEST_TARGET_BRANCH
CI_MERGE_REQUEST_ID
CI_MERGE_REQUEST_IID
CI_MERGE_REQUEST_REQUESTER_USER_ID

and so on, so we could manage what to do

This feature is crucial for us to guarantee a stable master, without having to build the resulting merge commit manually. So for that. And I like @Bastian's idea of additional environment variables as a temporary workaround.

Added ~182256 label

mentioned in issue #21991 (moved)

Changed title: Run build on merged code before merging (EE option?) → Run build on merged code before merging (Product?)

Added ~698938 label

@markpundsack is this gonna be a EE feature only? Really?

@Bastian:

Sid Sijbrandij wrote

If this is not contributed we might opt to make it a GitLab EE feature.

Unfortunately and AFAIK there is currently no PR from the community for this feature.
The company, I work for, would also like this feature. But as no one of us is a Ruby developer, and the unlikeliness of time to spend on open-source, nice-to-have features, you better not expect something coming from this direction.

Maybe, after implementation of this feature, I can persuade my company to get a EE license (as I would also like to have other EE features).
So for me, it seems legitimate to add this as an EE feature, if it isn't contributed by the community.

We are currently on community and will eventually end up on EE, but is there any chance of this feature making it in the community edition? This is pretty annoying that we're seeing long lived Merge Requests breaking destination branches.

Mentioned in issue #22989 (closed)

Added ~423634 label

/cc @markpundsack

we would need to trigger build for every commit that drops into master after merge request has been created

Ideally this would be automatic, but I suppose one could flip a switch and make this a more manual process. So, if the target branch has moved since the last build, then the UI would say e.g. "master has changed since the last build for this merge request. [Build again]"

On Fri Oct 7 16:21:59 2016 GMT+0200, Jonathon Reinhart wrote:

New comment for Issue 4176

https://gitlab.com/gitlab-org/gitlab-ce/issues/4176#note_16660583

Author: Jonathon Reinhart

we would need to trigger build for every commit that drops into master after merge request has been created

Ideally this would be automatic, but I suppose one could flip a switch and make this a more manual process. So, if the target branch has moved since the last build, then the UI would say e.g. "master has changed since the last build for this merge request. [Build again]"

I think this would defeat part of the purpose.

Perhaps you can enable that only for WIP merge requests? These tend not to be needed to be mergeable at every point in time.

Perhaps there are more situations you can think of to delay/don't do the automatic merge test.

I would have to agree with Ruben ... Long lived merge requests have a tendency to be either new features, but not for current state of master. Large features where all contributors can't work on it at the same time. Building on every commit just posts a bunch of failed builds on a feature branch. IMHO, a checkbox for every commit. A 'force build' button should the project master want a fresher build against the destination branch.

I think this is more a question of what's right for a particular workflow than trying to find a single, all-encompassing solution.

If you want master to always be stable and always pass tests, then builds should retrigger when something is merged into master. In this situation, master builds should never fail tests. This is best for teams with little resources, where a failure in master might not be addressed immediately. I know that at the small company where I work, this would be preferable.

If you're a larger project that sees many merges into master on a daily basis, it makes more sense to do point-in-time builds that are not retriggered from a merge into master; it's more efficient and you likely have the organisational capacity to fix any problems that occur on master. I imagine this is how the gitlab code base works.

IMO these are the two options that should be available, and they should be framed as such when selecting them as options.

I'm going to start hacking away at the code to try to implement this. I'm a bit new to rails, but I think I've got it figured out where in the code the MR refs are created. The one thing I haven't looked at yet is how a rebuild can be triggered. If anyone has any pointers on this, I'm all ears.

@squarebracket Hopefully some of my analysis from April is still valid, and will be of some use in your efforts: https://gitlab.com/jonathon-reinhart/gitlab-ce/issues/1

Removed ~13921 label

Hey I'm also looking to set up testing before or in response to a merge request being triggered. I was thinking though, if the main desire here is to ensure that some tests script run before changes are merged into another branch it may be an easier fix to build in a utility to disable merge requests until the branch reaches a particular step in its pipeline. Though this may not run automatically and would run on the whole branch rather than just the change as I understand it I think having a means of disabling the merge request button would allow users to require tests run before merging. I think this would also allow users to utilize the pipeline workflow already in place rather than setting up a whole new work flow for the merge requests themselves.

Feature request open for an option to disable merge requests: https://gitlab.com/gitlab-org/gitlab-ce/issues/2456

@MatthewRaison I think your problem is different to the one describe here. I think your problem is the following: Before branch B is merged into branch A (e.g. master), all tests have to run on the code in branch B before you can merge. The problem this issue tries to solve is different: Before branch B is merged into A, all test are run on the result of the (hypothetical) merge. This would ensure not only that the code in branch B works, but it also works when integrated back into branch A

@jzielke @Bastian
Really need this feature Can webhook with jenkins do this magic?

@yanceywang It can be done through Jenkins, with the GitLab plugin. You can attach the merge event hook as well and builds will run on merge requests, though note that it will still use the latest commit in the MR as the build commit, so if your Jenkins build takes a while to start then any earlier build-result for that commit will be used for build status of the MR.

Our current setup has a meta "Pipeline" stage that Jenkins reports as running the moment it receives the webhook, since the checkout and load of Jenkinsfile might otherwise take long enough for someone to be able to accept the MR without having a build run for it first.

** @ace13 I have just thought another way to trigger pipeline build when MR made .**

if you use Gitlab-CI and runner .Maybe You can add a webhook for MR events to trigger runner build . Any one know how to trigger runner using web hook ?

You can create a build trigger if you're using GitLab-CI, possibly hook it as a MR webhook depending on how your branches look.

The build trigger page even provides an example for doing a build from a webhook;

Use webhook

Add the following webhook to another project for Push and Tag push events. The project will be rebuilt at the corresponding event.
https://gitlab.example.com/api/v3/projects/PROJECT_ID/ref/REF_NAME/trigger/builds?token=TOKEN

If you need MR builds that test the merged state of the code though, then I'm not quite sure how to do it with GitLab-CI

removed ~123454 label

Is there any easy way to trigger build on merge requests automatically using gitlab-ci?

Just like github build pull requests before merge.

mentioned in issue #26638 (closed)

mentioned in issue #23015 (closed)

changed title from Run build on merged code before merging (Product?) to Run build on merged code before merging

added ~1464954 label

Triggering a new merge build every time something lands on master is not practical for a project the size of GitLab. The load on the runners would be roughly proportional to number of users with master privileges multiplied by the number of merge requests. This wastes runner time, because merges end up racing with each other, and the first one to pass invalidates all other tests.

The "train" solution is to form a queue of potential merges, and require each merge to pass the test suite before committing and starting the tests for the next merge candidate. This would likely grind a project like GitLab to a halt, because they merge a MR every 10 minutes and the test suite takes 2 hours to run. Every day they would increase the backlog of tests 3 more days.

A way to solve the backlog issue is to assume that all merge requests in the queue will pass, and optimistically test on top of them. When a MR is added to the merge queue, it is rebased onto the previous MR in the queue, and the test suite is started immediately. If a MR fails the test suite, it is discarded from the queue, it's status is set back top open. Any tests that were based on the failed MR are invalidated, and they are rebased on the updated queue.

The amount of work that would be invalidated by a failed MR is likely much less than the manual retries currently produced when a failing MR lands on master.

One way to think about this is that MRs are immediately merged into a shadow of the master branch.

If a MR fails:

It's shadow commit is reverted
It's status is set back to "open"
Everything after it in the queue is restarted

When a test on top of the queue passes:

The shadow commit is merged into the real master branch
The MR status is set to "merged"

I propose adding it as a project setting. Does this seem like a feature that would be suitable for CE?

A good first step that I would like to see is for gitlab to create a shadow merge to the target branch, and then provide access via a custom refspec. Perhaps 'refs/merge-requests/merged/<number>/head' to extend the existing merge-request refspec.

With this is place, I could make my jenkins multibranch pipeline build and test the merged result. Currently this is not possible due to limitations in the gitlab/jenkins plugin (https://github.com/jenkinsci/gitlab-plugin/issues/298)

Ideally, this would also mean that if everything went well, then the commit that shows up on master would have the same hash as the one that was tested. I am using Jenkins to build FPGA images. It is not uncommon to have build times of 12 hours for larger projects, and it would be nice if I didn't have to rebuild after the merge just because the commit hash changed.

@derrick.gibelyou I agree that GitLab needs to at least expose the "shadow merge". As I mentioned in duplicate issue #15122 (closed), GitHub already does this, and triggers CI on that commit. My proposal was in several stages:

GitLab has the "accept merge request" button, which means it has already tested the merge-ability of the MR head. It should simply store the result of this merge at refs/merge-requests/<number>/head.
Then, a build should be triggered for refs/merge-requests/<number>/head, and this build should be the red-light/green-light indicator shown on the MR page.
This is the step where everyone has a different opinion. What to do when master moves. For some projects, it makes sense to automatically re-build/test the other outstanding merges. As I mentioned above:

Ideally this would be automatic, but I suppose one could flip a switch and make this a more manual process. So, if the target branch has moved since the last build, then the UI would say e.g. "master has changed since the last build for this merge request. [Build again]"

@deckar01's proposal seems good-intentioned, but it sounds very complex.

I concur with @JonathonReinhart that 1 & 2 seem like widely agreeable improvements toward a solution to this issue, and 3 is where most of the debate is coming from. I think a first implementation that just provides the shadow merge commit ref and displays build status of this ref would be welcomed just as a show of momentum. If there is a merge conflict because the destination branch has moved ahead of the merge commit, we already have logic and UI to block the MR for that. And updating the source branch in that situation (by rebase, cherry-pic, or merge from destination branch) will already trigger the appropriate events for CI to rebuild so the only extra logic would be creating the new merge commit, updating the ref. I think GitLab has a really strong record of implementing some big new features in phases over recent releases and this seems like an ideal case for incremental implementations.

mentioned in issue #30439 (closed)

Maybe a "ready to be merged" element/switch could be added, that is activated by the person that submitted the MR and confirmed by the repo owner?

I imagine something like this:

MR "WIP: Feature X" is opened, some commits are added over time and comments/code review happens. CI builds only the branch of the MR.
After a while, the MR proponent locks in the current state of the MR with a "ready to be merged" button. The repo owner (potentially after checking some requirements like CLAs, test coverage, moon phase...) then names a branch where this would be merged, default would be master and confirms. CI now continuously builds the merge with that branch (automatically if there was no merge build in the last n minutes - fast moving projects or free services would set n higher than 0). The merge can now happen either automatically, if CI succeeds (defaults to disabled) or as it currently happens, after manual confirmation by the repo owner.
If the submitter still pushes further commits after marking the MR as "ready to be merged", these are not considered in the merge, the MR state goes back to "WIP" and CI only builds the branch, not the merge.
Once every n minutes since the last merge build, a MR submitter can manually request a merge build with a target branch - to look for merge conflicts etc.

This scheme would ensure that there are rather few merge builds compared to branch builds if the target branch is busy, while enabling as much automation as possible.

mentioned in issue #17265 (moved)

Hello guys, I built something which solved the problem for us, maybe it can help someone else while Gitlab team is working on the feature: https://github.com/MelwinKfr/gitlab-webhook-translator

In order to reduce our master branch specs failures on merging MRs into master, can we prioritize this feature proposal?

mentioned in issue #33400 (closed)

mentioned in issue #33674 (moved)

With https://gitlab.com/gitlab-org/gitlab-ce/issues/4176#note_13717683, I could do a diff on the file names and only run unit tests on the changed files.

mentioned in issue #21947 (closed)

Also see https://smarketshq.com/marge-bot-for-gitlab-keeps-master-always-green-6070e9d248df

... we have all our accepted merge-requests effectively going through a [priority queue] and processed one by one.

https://smarketshq.com/marge-bot-for-gitlab-keeps-master-always-green-6070e9d248df

That is a suitable strategy for most projects, however, projects the size of GitLab CE need builds to run in parallel or they will form a growing backlog.

When a MR is added to the merge queue, it is rebased onto the previous MR in the queue, and the test suite is started immediately.

https://gitlab.com/gitlab-org/gitlab-ce/issues/4176#note_25628275

You need to proactively rebase on the previous item in the queue to allow builds to continue in parallel.

I think having the option to use a simple auto-rebase merge mode would be a good start. But, this issue isn't just a bug, it's an incorrect design of continuous integration. @markpundsack I don't think this should be ~"EE Premium".

@deckar01 Yes, agreed -- for realy large scale you want to pro-actively rebase to be able to speculatively run builds. But as you also point out for most use cases, this is not needed, so we haven't provided that functionality yet, although we might find we want it in the future (it's not conceptually hard, just extra work). One thing that would be nice BTW, and I should probably file a new issue for this, is if gitlab would be a bit smarter about caching CI results. Currently it re-runs when you just change the commit messages (but the code in fact stays the same). It would be nice to compute the hash over the working tree and cache based on that.

mentioned in issue #35261 (closed)

marked this issue as related to #35261 (closed)

More support for this feature: https://gitlab.zendesk.com/agent/tickets/80260 (GitLab internal only)

changed title from Run build on merged code before merging to Test merge with master: run build on merged code before merging

Inspired by a large organization I wrote up the release train process in the description:

As a user when I merge a feature branch I want to make sure that master is still green after that.
To be 100% sure it requires testing the merge of the feature with master, but master changes all the time.
Some repositories get one merge a minute, the build+test time is longer than the time a merge can take.
This calls for a release train, in which merges are queued in a sequence.

How the feature works:

Enable 'Merge when pipeline succeeds' https://docs.gitlab.com/ce/user/project/merge_requests/merge_when_pipeline_succeeds.html
Enable 'Merge when master succeeds' or 'Release train', a new function
All merge requests are tested as a feature branch on every commit, like you expect.
If the feature branch is green the merge request has a button 'Merge when master succeeds' or 'Add as the 3th MR to the merge train'
When you click 'Merge when master succeeds' on the first MR the tests start running for a merge with master
If there is another MR where this button was already pushed and that is already running tests the second MR will test the merge with this MR's merge with master
Repeat for all other MR's, they are sequenced based on when the button was pressed and form a release train
If a test fails on an MR it is no longer merged and all following MRs are retested.

changed title from Test merge with master: run build on merged code before merging to Merge train/Merge when master succeeds: run build on merged code before merging

changed the description

Once I wrote a sketch for this feature. Probably this is similar with other implications.

It could be implemented. Although the curial problem is underlying on https://gitlab.com/gitlab-org/gitlab-ce/issues/4176#note_35184286.

changed the description

marked this issue as related to #27337 (moved)

@sytses Good proposal. A more boring first iteration might be to skip the "train" aspect. It will fail for the large-org circumstances you mentioned, but would work fine for smaller orgs and let us explore the space more cheaply. e.g. no overhead of managing a queue. This is how marge-bot works (https://gitlab.com/gitlab-org/gitlab-ce/issues/35261) and at least some people seem to like it.

Also, if combined with https://gitlab.com/gitlab-org/gitlab-ce/issues/32741, it could be just as wall-clock efficient, perhaps even more so (because failed pipelines won't have delayed downstream MR pipelines). It's more compute intensive though.

@markpundsack cool! I love the iterative approach

customer https://gitlab.my.salesforce.com/0016100000fyZHW has indicated this feature would make their processes much simpler. Currently achieving this through a custom slack app with a whole lotta magic

Cool @jdrumtra, can we have some additional details about the custom slack app approach?

@victorwu and I spoke with them this morning on discussion related topics. I think they would be open to talking about their implementation @bikebilly

Will direct to this issue and ask if they can spend some time to discuss.

mentioned in issue #37094 (closed)

changed title from Merge train/Merge when master succeeds: run build on merged code before merging to Merge train/Release train/Merge when master succeeds: run build on merged code before merging

added customer label

A customer is interested in the feature. cc @markpundsack

Multiple customers at this point I'd say (my employer being another interested EE customer). I still think simply adding # 1 and # 2 items from @JonathonReinhart comment 26768258 would be the simplest and most flexible option. Especially given how popular Jenkins/Bamboo/Teamcity still are (ie these two small improvements open the door for any CI system to build the merge result and post the status). Maybe add those two to CE and leave all the rest of the build/release train magic that's being discussed as EE-only.

marked this issue as related to #15122 (closed)

Is this feature planned for development? When you estimate to release it?

We're currently using GitLab Enterprise Edition 9.5.5-ee. Thanks :)

I wanted to share an analogy. The release train is like branch prediction on a CPU. As long as it goes well it can speed things up. When a build fails (taking another branch) the work has to be discarded.

Mockup for 2018 product vision presentation:

@markpundsack this view uses the first revised MR widget concept of the idea coming forth out of our discussion from https://gitlab.com/gitlab-org/gitlab-ce/issues/37479 so there is a clear separation of pipelines used to check the code of the mr and pipeline which will check the merging.

Merge is in cue:

Merge is in active cue:

Merge cue view:

Thanks @dimitrieh! I'll leave deeper comments for later. For now:

I wonder if this should start with a page showing that there are X MRs in queue before this one, and a button to add this MR to the release train. It's hard to grok the first image.
Likewise, in the text of the first image, it says it'll be tested after some pipeline #, but it's not really clear that it's in queue. Maybe it should say it is #Y in queue. Maybe even use the words "release train" somewhere, but could just say it's in queue to be tested and released. We don't really use those words though, so it might be more realistic to say the pipeline is in queue to run.
The merge queue tab is a good idea, but is confusing because it's showing WIP (which would never be merged) and items with pipelines in progress, when presumably only one would ever be in progress at the same time.
The new tabbed interface confuses the issue and people will focus on that rather than the feature itself.
Why would there be more than one pipeline merging at the same time?

@markpundsack i thought mr's were automatically placed in the merge train. And if a pipeline has started to test out the mr with the first X mr's.. it cannot be added to the train anymore.

Also thought it was not possible to pre schedule the pipeline.. as it is not certain what the code will be run on (which exact mr's), until the last moment.

Why would there be more than one pipeline merging at the same time?

How so? the cueue just shows the one pipeline which tests the current merge train.

Updated temp designs:

@dimitrieh The idea behind the merge train is that you do review like normal, with peer review, automated CI tests, etc., all done in the branch, but when you're ready to press the merge button, you want to run one more set of tests, but this time on the post-merge code.

So when you click Merge, if nothing else is in progress, it starts one more test on the merged version, and if it still passes, then the merge is saved or committed to the project's master.

But if another merge was already in progress, you don't want to start your final merge and test, because you don't know if the previous merges are going to succeed, so you don't know exactly what you should be testing against. Therefore you wait, without running the pipeline at all, until it's your time in the queue. Therefore, there will only be one merge train pipeline running at a time. The rest are just waiting.

MRs that aren't ready to be merged run normal pipelines on their branches, but that's outside of the release train queue.

I think it would be valuable to know that my MR would be queued if I press the Merge button, with how many are in the queue, or even just the button could say "Add to merge queue". But since it's probably not worth updating that in realtime, I can see dropping it. After pressing Merge, it would just tell me it's queued up.

Rather than say the merge is "in progress" and also "in queue" at the same time, I'd rather just say that it is #5 (closed) in the release queue. I don't see the value in showing me which pipeline it comes after. If anything the words "release queue" could be a link to the queue, where perhaps I might want to cancel some, or rearrange, or just watch as it proceeds. But telling me another pipeline ID doesn't seem to add value.

The last image still shows pipeline status on the right hand side. Since anything in the queue is by definition not running, that makes no sense and should be removed.

Small thing, but rather than showing when they were updated, how about showing how long ago they were queued? And make sure they're in reverse order, since oldest one would be at the top.

@markpundsack

Next batch, with improvements (these will need to be updated in the presentation right?)

As for the merge queue view.. i can update later.. have a question (Also related to https://gitlab.com/gitlab-org/gitlab-ce/issues/4176#note_42507333):

@markpundsack

mentioned in commit gitlab-design@ac649609

How so? the cueue just shows the one pipeline which tests the current merge train.

@dimitrieh The mockup shows Merging (6).

The mockup shows Merging (6).

@markpundsack as you can see, it shows 6 merge requests in the cue... with the current test running at the top.

assigned to @dimitrieh

changed the description

mentioned in issue #38862 (closed)

This impacts us tremendously when a large flurry of merge requests all come in at the same time. We've begun to use marge-bot, but would prefer to group them together but still retain the merge request for the group.

Do we have an estimated ship date for this yet?

@chill9 not defined yet, as it is a very complex story. For now it is marked %"Next 6-12 months" that means we want it and we are already working to fully define what it will be.

mentioned in issue #40321 (moved)

We're still very much in need of this... We are an ee+ customer. We have a live example of why this would be extremely helpful today if anyone from Gitlab wants to appear.in with us. Maybe we can have recommendations as a stop gap?

@tthorslund @k33g @rgronkowski - is this something you can help us out with?

oh wow, this issue is closed?! It was marked for the %"Next 6-12 months" earlier... What's up with this @bikebilly ?

@pvdissel Sorry it's confusing, but it's moved to https://gitlab.com/gitlab-org/gitlab-ee/issues/3709 rather than closed. You could see the note at https://gitlab.com/gitlab-org/gitlab-ce/issues/4176#note_43001643

@chill9 Can we speak a little later today? I left you a couple of vms as well.

@godfat thanks, subscribed to the new issue :) Though, sad and concerning to hear it will be EE+ only.. as said earlier in this thread, the current way of CI is an incorrect implementation of continuous integration to begin with. But ok, will follow along in the EE issue

mentioned in issue #55039 (moved)

mentioned in issue ux-research#253

Merge train/Release train/Merge when master succeeds: run build on merged code before merging

Designs

Child items ...

Activity

Solution

Updated temp designs:

Merge train/Release train/Merge when master succeeds: run build on merged code before merging

Relates to

Activity

Solution

Updated temp designs: