Automate Quarantining end-to-end test cases (v1)

• Define the eligibility criteria for a testcase to be quarantined (3)
  • Automated test fails 3 amount of times in a given environment
• Once defined, fetch data required to quarantine the test case (these might already exist)
  • Fetch the last failure issue
  • Any existing bugs.
  • Environment it consistently failed in
• v1 Iteration: By default set the quarantine type as investigating
  • quarantine: { issue: ‘[ISSUE]’, type: :investigating}
• Scalability - Add Ability to apply to any GitLab project
  • Create new MVC project (gem)
  • Add Confiner gem and rules (!77809 - merged)
  • Add auto_quarantine job (!78092 - merged)
• Open an MR to quarantine the test case with the above data
  • v1 Iteration: Bot doesn't mention anyone on the MR.
  • v2 iteration: Automatically assign the MR to the on call pipeline DRI for that day.
  • v3 Iteration: Add this MR to our Pipeline triage weekly report : gitlab-org/quality/pipeline-triage#104 (closed)
• Define a process for the MR to be reviewed by SETs in a timely fashion.
  • Update our handbook to add guidelines for the automated test failures / automating quarantining process (gitlab-com/www-gitlab-com!96758 (merged)).
• Conduct an AMA/show and tell in Bi-Weekly Architectural meeting.