[CI] Require manual action to deploy to CI clusters

Summary

Note: this is just an idea, and will likely require some iteration or we'll ditch it entirely.

TL;DR: let's consider the following flow:

  • When updates to a branch are pushed, only trigger jobs that do not touch the CI clusters
  • Consolidate the jobs that do not touch CI clusters so they run in parallel when possible
  • When desired, the MR author can manually trigger the full tests in the CI clusters

Why?

  • This reduces load on CI clusters, which has been causing numerous failures
  • Not every commit warrants a full test suite in CI
  • The feedback cycle would be shorter/faster
  • We maintain the ability to test changes end-to-end that we have today

Context

From #230 (comment 662195614):

Had a good chat with @dmakovey on this topic today. Some notes:

Maybe it doesn't make sense (at the moment at least) to deploy to CI and run QA on every commit to an MR branch. Instead, the MR author could manually trigger these stages on a specific pipeline when a full test is desired.

This would:

  • free up a lot of capacity in our CI clusters
  • reduce false negatives in the pipelines (many failures today are just resource contention / timeouts)
  • speed up the feedback cycle because we could run the tests and Docker build all at the same time
  • remove the usage of CI time/resources for commits that don't need functional/QA tests (like documentation changes, etc - we could address this with docs-* branch prefix/suffix of course, as we do with Charts CI)

This means that the MR author would be responsible for running at least one full QA pipeline prior to opening an MR for review.

Still thinking over the implications here but wanted to share in case anyone has feedback here in the meantime 👍

diff --git a/.gitlab-ci-templates.yml b/.gitlab-ci-templates.yml
index b9c1adc..f2af20f 100644
--- a/.gitlab-ci-templates.yml
+++ b/.gitlab-ci-templates.yml
@@ -18,6 +18,7 @@
 .review_template:
   extends: .cache
   stage: review
+  when: manual
   variables:
     CLEANUP: "no"
   script:
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 7c6f33a..708325f 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -59,6 +59,7 @@ pull_charts:
 lint_code:
   extends: .cache
   stage: test
+  needs: []
   image: registry.gitlab.com/gitlab-org/gitlab-build-images:golangci-lint-alpine
   script: golangci-lint run --out-format code-climate | tee gl-code-quality-report.json | jq -r '.[] | "\(.location.path):\(.location.lines.begin) \(.description)"'
   artifacts:
@@ -103,6 +104,7 @@ slow_unit_tests:
 .docker_build_job:
   extends: .cache
   stage: release
+  needs: ["pull_charts"]
   image: docker:latest
   services:
     - docker:dind
Edited by Mitchell Nielsen