Fix the 'review-deploy' job when RA deployment is manual

What does this MR do?

1. Always make the review-deploy job manual.
2. Move review-build-cng to the review stage so that review-build-cng and review-deploy can be started without the other being done first.
3. Always try to trigger review-deploy at the end of review-build-cng.
4. In review-deploy:
   1. If review-build-cng is not started, start it.
   2. Wait for review-build-cng to finish.
   3. Proceed to deployment.
   4. At the end, play review-qa-smoke and review-performance.
5. In review-qa-smoke and review-performance, wait for review-deploy to be finished, then download its environment_url.txt artifact.

Note that #31264 (closed) would allow to avoid most if not all of these workarounds.

Validation

With no frontend change, no deploy

1. The review-build-cng and review-deploy jobs are playable: https://gitlab.com/gitlab-org/gitlab/pipelines/131907868
2. Pipeline shouldn't be stuck in the Running state

With no frontend change, playing review-deploy first

1. Play review-deploy: https://gitlab.com/gitlab-org/gitlab/-/jobs/500096800
   1. Expect review-build-cng to be played first
   2. Expect to wait for review-build-cng to be done before proceeding to the deploy
   3. Expect review-qa-smoke and review-performance to be played at the end of review-deploy
2. Expect review-qa-smoke and review-performance to pass

With no frontend change, playing review-build-cng first

1. Play review-build-cng: https://gitlab.com/gitlab-org/gitlab/-/jobs/500543494
   1. Expect review-deploy to be played at the end of review-build-cng
      1. Expect review-qa-smoke and review-performance to be played at the end of review-deploy
2. Expect review-qa-smoke and review-performance to pass

With a frontend change (so that Review App deployment is automatic

1. Expect review-build-cng to start automatically: https://gitlab.com/gitlab-org/gitlab/-/jobs/492379779
   1. Expect review-deploy to be played at the end of review-build-cng
2. Expect review-qa-smoke and review-performance to start automatically after review-deploy is done
3. Expect review-qa-smoke and review-performance to pass

Closes #212354 (closed).

.gitlab/ci/review.gitlab-ci.yml

@@ -64,7 +64,7 @@ review-build-cng:
     - .default-retry
     - .review:rules:mr-and-schedule-auto-if-frontend-manual-otherwise
   image: ruby:2.6-alpine
-  stage: review-prepare
+  stage: review
   before_script:
     - source scripts/utils.sh
     - install_api_client_dependencies_with_apk
@@ -76,7 +76,7 @@ review-build-cng:
     - BUILD_TRIGGER_TOKEN=$REVIEW_APPS_BUILD_TRIGGER_TOKEN ./scripts/trigger-build cng
     # When the job is manual, review-deploy is also manual and we don't want people
     # to have to manually start the jobs in sequence, so we do it for them.
-    - '[ -z $CI_JOB_MANUAL ] || play_job "review-deploy"'
+    - play_job "review-deploy"
 
 .review-workflow-base:
   extends:
@@ -96,11 +96,12 @@ review-build-cng:
 review-deploy:
   extends:
     - .review-workflow-base
-    - .review:rules:mr-and-schedule-auto-if-frontend-manual-otherwise
+    - .review:rules:review-deploy-manual
   stage: review
-  dependencies: []
+  needs:
+    - job: gitlab:assets:compile pull-cache
+      artifacts: false
   resource_group: "review/${CI_COMMIT_REF_NAME}"
-  allow_failure: true
   before_script:
     - '[[ ! -d "ee/" ]] || export GITLAB_EDITION="ee"'
     - export GITLAB_SHELL_VERSION=$(<GITLAB_SHELL_VERSION)
@@ -110,6 +111,9 @@ review-deploy:
     - source ./scripts/utils.sh
     - install_api_client_dependencies_with_apk
     - source scripts/review_apps/review-apps.sh
+    # We don't want people to have to manually start the jobs in sequence, so we do it for them.
+    - '[ -z $CI_JOB_MANUAL ] || play_job "review-build-cng"'
+    - '[ -z $CI_JOB_MANUAL ] || wait_for_job_to_be_done "review-build-cng"'
   script:
     - check_kube_domain
     - ensure_namespace
@@ -118,8 +122,7 @@ review-deploy:
     - download_chart
     - date
     - deploy || (display_deployment_debug && exit 1)
-    # When the job is manual, review-qa-smoke is also manual and we don't want people
-    # to have to manually start the jobs in sequence, so we do it for them.
+    # We don't want people to have to manually start the jobs in sequence, so we do it for them.
     - '[ -z $CI_JOB_MANUAL ] || play_job "review-qa-smoke"'
     - '[ -z $CI_JOB_MANUAL ] || play_job "review-performance"'
   artifacts:
@@ -162,9 +165,10 @@ review-stop:
 .review-qa-base:
   extends: .review-docker
   stage: qa
-  # This is needed so that manual jobs with needs don't block the pipeline.
-  # See https://gitlab.com/gitlab-org/gitlab/-/issues/199979.
-  dependencies: ["review-deploy"]
+  # Hack so that the jobs can be played by `review-deploy` (before it's finished).
+  needs:
+    - job: review-build-cng
+      artifacts: false
   allow_failure: true
   variables:
     QA_ARTIFACTS_DIR: "${CI_PROJECT_DIR}/qa"
@@ -177,13 +181,20 @@ review-stop:
     GITHUB_ACCESS_TOKEN: "${REVIEW_APPS_QA_GITHUB_ACCESS_TOKEN}"
     EE_LICENSE: "${REVIEW_APPS_EE_LICENSE}"
   before_script:
+    - source scripts/utils.sh
+    - install_api_client_dependencies_with_apk
     - '[[ ! -d "ee/" ]] || export GITLAB_EDITION="ee"'
     - export QA_IMAGE="${CI_REGISTRY}/${CI_PROJECT_PATH}/gitlab/gitlab-${GITLAB_EDITION}-qa:${CI_COMMIT_REF_SLUG}"
+    # If this job is played from `review-deploy`, it would start before `review-deploy`
+    # is finished, leading to no artifact being downloaded.
+    # Thus, we need to:
+    # 1. Wait for `review-deploy` to finish so that artifact exists
+    # 2. Fetch the artifact manually
+    - wait_for_job_to_be_done "review-deploy"
+    - retrieve_artifact "review-deploy" "environment_url.txt"
     - export CI_ENVIRONMENT_URL="$(cat environment_url.txt)"
     - echo "${CI_ENVIRONMENT_URL}"
     - echo "${QA_IMAGE}"
-    - source scripts/utils.sh
-    - install_api_client_dependencies_with_apk
     - gem install gitlab-qa --no-document ${GITLAB_QA_VERSION:+ --version ${GITLAB_QA_VERSION}}
   artifacts:
     paths:
@@ -194,7 +205,7 @@ review-stop:
 review-qa-smoke:
   extends:
     - .review-qa-base
-    - .review:rules:mr-only-auto-if-frontend-manual-otherwise
+    - .review:rules:review-deploy-manual
   script:
     - gitlab-qa Test::Instance::Smoke "${QA_IMAGE}" "${CI_ENVIRONMENT_URL}"
 
@@ -210,20 +221,12 @@ review-qa-all:
 
 review-performance:
   extends:
-    - .review-docker
-    - .review:rules:mr-and-schedule-auto-if-frontend-manual-otherwise
-  stage: qa
-  # This is needed so that manual jobs with needs don't block the pipeline.
-  # See https://gitlab.com/gitlab-org/gitlab/-/issues/199979.
-  dependencies: ["review-deploy"]
-  allow_failure: true
-  before_script:
-    - export CI_ENVIRONMENT_URL="$(cat environment_url.txt)"
-    - echo "${CI_ENVIRONMENT_URL}"
+    - .review-qa-base
+    - .review:rules:review-deploy-manual
+  script:
     - mkdir -p gitlab-exporter
     - wget -O ./gitlab-exporter/index.js https://gitlab.com/gitlab-org/gl-performance/raw/master/index.js
     - mkdir -p sitespeed-results
-  script:
     - docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:6.3.1 --plugins.add ./gitlab-exporter --outputFolder sitespeed-results "${CI_ENVIRONMENT_URL}"
   after_script:
     - mv sitespeed-results/data/performance.json performance.json