Skip to content

Automatically merge CE master into EE master

Yorick Peterse requested to merge merge-train into master

What does this MR do?

This sets up GitLab CI to automatically push CE master changes into EE master, or revert them if the changes cause merge conflicts. The CI configuration contains a single job to do this: merge:master. This job is executed for every push to master, and periodically using a CI schedule.

The periodic job is necessary because incremental jobs may not be able to revert commits if newly added commits depend on these commits. By re-running the job periodically (including all changes since a large enough time frame), we can ensure that such commits are also reverted (if they still conflict at that time).

The job runs in its own "merge" stage, after the build and prepare stages, but before running the tests. This ensures that randomly failing tests won't prevent code from being merged into EE. Running the stage after the "prepare" stage reduces the chances of the job reverting CE changes just because it ran before a corresponding EE MR was merged into EE master.

The container used by this CI job is built daily in https://gitlab.com/gitlab-org/merge-train/.

Example CI output

Running with gitlab-runner 11.5.0-rc1 (e900028d)
  on docker-auto-scale 0277ea0f
Using Docker executor with image registry.gitlab.com/gitlab-org/merge-train ...
Pulling docker image registry.gitlab.com/gitlab-org/merge-train ...
Using docker image sha256:994868a307623150907ad66a0c491ca6dd866f1d9d73421b17be5adf816df2a9 for registry.gitlab.com/gitlab-org/merge-train ...
Running on runner-0277ea0f-project-9683125-concurrent-0 via runner-0277ea0f-srm-1543511992-95e93c87...
Cloning repository for master with git depth set to 20...
Cloning into '/builds/yorickpeterse/gitlab-ce'...
Checking out f1f70328 as master...
Skipping Git submodules setup
Checking cache for merge-1...
FATAL: file does not exist                         
Failed to extract cache
$ scripts/merge-train
Agent pid 12
Identity added: /root/.ssh/merge_train (automatic merge)
Cloning yorickpeterse/gitlab-ee
Warning: Permanently added the ECDSA host key for IP address '35.231.145.151' to the list of known hosts.
I, [2018-11-29T17:22:43.635210 #19]  INFO -- : Fetching master from remote merge-train-source
I, [2018-11-29T17:22:44.320720 #19]  INFO -- : 1 commits need to be merged
I, [2018-11-29T17:22:44.485251 #19]  INFO -- : f1f7032826336d527e89386fb5a4081151b3199a was merged successfully
I, [2018-11-29T17:22:44.485666 #19]  INFO -- : Pushing changes to master
I, [2018-11-29T17:22:45.983972 #19]  INFO -- : Pushed changes to master
Creating cache merge-1...
gitlab-ee: found 22784 matching files              
Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/9683125/merge-1 
Created cache
Job succeeded

Without the cache in place, the job will take at least 2 minutes and 50 seconds (ish). With a cache in place it takes at least one minute less to run.

What are the relevant issue numbers?

https://gitlab.com/gitlab-org/release/framework/issues/49

TODO

  • Set up EE MR for these changes: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8657
  • Store the SSH public and private key as a CI variable
  • Put the SSH public and private key in 1Password
  • Add the public key to GitLab EE
  • Create an API token attached to @gitlab-bot, and store it as a CI variable in CE
  • Stop the periodic "CE upstream" job, so it doesn't conflict in any way
  • Update any documentation mentioning the "CE upstream" job so that it covers the new setup
  • Announce on the company/backend call
  • Add to the engineering week in review document

Does this MR meet the acceptance criteria?

Edited by Rémy Coutable

Merge request reports