Build cascading train refs for parallel execution of Pipelines for merge trains

What does this MR do?

As we supported arbitrary refs creation in the previous merge request https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/14298, we can now create cascading merge refs in refs/train to run Pipelines for merge trains in parallel.

In this iteration, we support up-to 4 concurrent pipeline runs. Here is the overview of the process flow:

1. User adds MR1, MR2, MR3, MR4 and MR5 to a train.
2. The system creates the train ref refs/merge-requests/1/train from mr1.diff_head_sha and mr1.target_branch_ref.
3. The system creates a pipeline run on refs/merge-requests/1/train.
4. The system creates the train ref refs/merge-requests/2/train from mr2.diff_head_sha and mr1.train_ref_path.
5. The system creates a pipeline run on refs/merge-requests/2/train.
6. The system creates the train ref refs/merge-requests/3/train from mr3.diff_head_sha and mr2.train_ref_path.
7. The system creates a pipeline run on refs/merge-requests/3/train.
8. The system creates the train ref refs/merge-requests/4/train from mr4.diff_head_sha and mr3.train_ref_path.
9. The system creates a pipeline run on refs/merge-requests/4/train.
10. The system does not create any train refs refs/merge-requests/5/train because it's out of concurrency factor.
11. The system does not create any pipelines on refs/merge-requests/5/train because it's out of concurrency factor.

Related https://gitlab.com/gitlab-org/gitlab-ee/issues/11222 gitaly!1210 (merged)

Feature Flag

This behavior can be reverted by Feature.disable(:merge_trains_parallel_pipelines). After it's reverted, pipeline runs one at a time.

Extra datapoint

Today the number of projects which enables merge trains is still very low.

gitlabhq_production=> select COUNT(*) from project_ci_cd_settings WHERE merge_trains_enabled = true;
 count 
-------
    28
(1 row)

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• Documentation created/updated or follow-up review issue created
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• Separation of EE specific content

Performance and testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• [-] Tested in all supported browsers

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

• [-] Label as security and @ mention @gitlab-com/gl-security/appsec
• [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• [-] Security reports checked/validated by a reviewer from the AppSec team

ee/app/models/merge_train.rb

@@ -41,12 +41,20 @@ def all_next
     self.class.all_in_train(merge_request).where('merge_trains.id > ?', id)
   end
 
+  def all_prev
+    self.class.all_in_train(merge_request).where('merge_trains.id < ?', id)
+  end
+
   def next
     all_next.first
   end
 
+  def prev
+    all_prev.last
+  end
+
   def index
-    self.class.all_in_train(merge_request).where('merge_trains.id < ?', id).count
+    all_prev.count
   end
 
   def first_in_train?
@@ -54,6 +62,6 @@ def first_in_train?
   end
 
   def follower_in_train?
-    self.class.all_in_train(merge_request).where('merge_trains.id < ?', id).exists?
+    all_prev.exists?
   end
 end