Initial implementation of the merge train to automatically merge or revert CE changes

This adds the code necessary to get the merge train going (choo choo), as described in https://gitlab.com/gitlab-org/release/framework/issues/49.

The merge train will allow us to automatically merge changes from CE branches into their EE counterparts. Changes that can not be merged automatically are instead reverted, which can then later be reinstated using a merge request (also generated by the merge train). This should eventually remove the need for the periodic CE upstream merge requests, drastically reducing the time it takes to get CE changes into EE.

The Merge Train does not create an EE merge request based on a CE merge request to resolve any conflicts, you still need to do this yourself.

Commits are merged by generating patch files for them using git format-patch, which are then applied using git am --3way. A simple git merge usually generates way too many conflicts, even for unrelated files. It also conflicts when the commit has already been merged, but it doesn't explicitly tell us the conflict was due to the changes already being applied. git am --3way on the other hand will produce different output for conflicts versus already applied changes, making it much easier to determine what needs to be reverted and what was already applied. git am --3way does not produce merge commits in the target repository, so we'll start seeing fewer of those once the merge train is running.

Examples

Commit that conflicts with EE	Reverted in CE master	CE MR to reinstate changes

Hooking it up in GitLab CE

I hooked this up in a fork of CE (https://gitlab.com/yorickpeterse/gitlab-ce) using the following GitLab CI job:

---
merge:
  image: registry.gitlab.com/gitlab-org/merge-train
  before_script:
    - mkdir -p ~/.ssh
    - echo "$MERGE_SSH_PUBLIC_KEY" > ~/.ssh/id_id_ed25519.pub
    - echo "$MERGE_SSH_PRIVATE_KEY" > ~/.ssh/id_id_ed25519
    - chmod 700 ~/.ssh
    - chmod 644 ~/.ssh/id_id_ed25519.pub
    - chmod 600 ~/.ssh/id_id_ed25519
    - ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
    - eval "$(ssh-agent -s)"
    - ssh-add ~/.ssh/id_id_ed25519
    - git clone --depth=1 git@gitlab.com:yorickpeterse/gitlab-ee.git ~/gitlab-ee
  script:
    - cd /app
    - bundle exec /app/bin/merge-train "$CI_PROJECT_DIR" ~/gitlab-ee --source-name yorickpeterse/gitlab-ce

When running, this would then produce the following build output:

Running with gitlab-runner 11.5.0-rc1 (e900028d)
  on docker-auto-scale 72989761
Using Docker executor with image registry.gitlab.com/gitlab-org/merge-train ...
Pulling docker image registry.gitlab.com/gitlab-org/merge-train ...
Using docker image sha256:91791713cb6365542b92dc7139c6a5e94aff20487f0a876af533ce8b780732b2 for registry.gitlab.com/gitlab-org/merge-train ...
Running on runner-72989761-project-9491424-concurrent-0 via runner-72989761-srm-1543327861-281f30bc...
Cloning repository...
Cloning into '/builds/yorickpeterse/gitlab-ce'...
Checking out 316c757b as master...
Skipping Git submodules setup
$ mkdir -p ~/.ssh
$ echo "$MERGE_SSH_PUBLIC_KEY" > ~/.ssh/id_id_ed25519.pub
$ echo "$MERGE_SSH_PRIVATE_KEY" > ~/.ssh/id_id_ed25519
$ chmod 700 ~/.ssh
$ chmod 644 ~/.ssh/id_id_ed25519.pub
$ chmod 600 ~/.ssh/id_id_ed25519
$ ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
# gitlab.com:22 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.6
# gitlab.com:22 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.6
# gitlab.com:22 SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.6
$ eval "$(ssh-agent -s)"
Agent pid 14
$ ssh-add ~/.ssh/id_id_ed25519
Identity added: /root/.ssh/id_id_ed25519 (automatic merge)
$ git clone --depth=1 git@gitlab.com:yorickpeterse/gitlab-ee.git ~/gitlab-ee
Cloning into '/root/gitlab-ee'...
Warning: Permanently added the ECDSA host key for IP address '35.231.145.151' to the list of known hosts.
Checking out files:  66% (12595/19036)   
Checking out files:  67% (12755/19036)   
Checking out files:  68% (12945/19036)   
Checking out files:  69% (13135/19036)   
Checking out files:  70% (13326/19036)   
Checking out files:  71% (13516/19036)   
Checking out files:  72% (13706/19036)   
Checking out files:  73% (13897/19036)   
Checking out files:  74% (14087/19036)   
Checking out files:  75% (14277/19036)   
Checking out files:  76% (14468/19036)   
Checking out files:  77% (14658/19036)   
Checking out files:  78% (14849/19036)   
Checking out files:  79% (15039/19036)   
Checking out files:  80% (15229/19036)   
Checking out files:  81% (15420/19036)   
Checking out files:  82% (15610/19036)   
Checking out files:  83% (15800/19036)   
Checking out files:  84% (15991/19036)   
Checking out files:  85% (16181/19036)   
Checking out files:  86% (16371/19036)   
Checking out files:  87% (16562/19036)   
Checking out files:  88% (16752/19036)   
Checking out files:  89% (16943/19036)   
Checking out files:  90% (17133/19036)   
Checking out files:  91% (17323/19036)   
Checking out files:  92% (17514/19036)   
Checking out files:  93% (17704/19036)   
Checking out files:  94% (17894/19036)   
Checking out files:  95% (18085/19036)   
Checking out files:  96% (18275/19036)   
Checking out files:  97% (18465/19036)   
Checking out files:  98% (18656/19036)   
Checking out files:  99% (18846/19036)   
Checking out files: 100% (19036/19036)   
Checking out files: 100% (19036/19036), done.
$ cd /app
$ bundle exec /app/bin/merge-train "$CI_PROJECT_DIR" ~/gitlab-ee --source-name yorickpeterse/gitlab-ce
I, [2018-11-27T14:13:58.797785 #21]  INFO -- : Fetching master from remote merge-train-source
I, [2018-11-27T14:15:10.263375 #21]  INFO -- : 1 commits need to be merged
I, [2018-11-27T14:15:10.641022 #21]  INFO -- : 316c757b2dcf522903d8c130bdcff04e61f5e0f4 can not be merged
E, [2018-11-27T14:15:11.083919 #21] ERROR -- : Reverting 316c757b2dcf522903d8c130bdcff04e61f5e0f4 on branch master for project yorickpeterse/gitlab-ce
Job succeeded

Here somewhat ironically it reverts some of my changes to .gitlab-ci.yml, but that's fine for the sake of this example.