Test adding a new release environment automatically when releasing 16.11.0

Testing on 16.10.1 - Failed

NOTE: The following test failed, because the feature flag release_environment was not activated. We will redo it in 16.11.0

After finishing #19580 (closed) , we need to do an E2E test for the automation of creating a new release environment. At the time of creating this issue, 16.10.0 was tagged and the release environment for 16.10 was created manually since the MR for #19580 (closed) ( gitlab-org/release-tools!2876 (merged)) wasn't merged yet (actually I used the MR, but ran it locally to create the new environment).

Thus, we can test the automation when releasing 16.10.2, since the automation can happen at any version tagging. However, some manual steps are required. The step-by-step is as below:

• Inform the Release Managers about the activities in this issue.
• Delete the environment 16.10 (a MR is ready https://gitlab.com/gitlab-com/gl-infra/release-environments/-/merge_requests/184).
• Note that after deleting the environment, all new deployments to the release environment (by merging to the stable branch 16-10-stable-ee) will fail. Thus, we try to do it as near to the tagging date as possible. I set the Due date of this issue to 2024-04-09, one day before the release date.
• Ask the RMs to inform when they tag the new release.
• Follow the release:tag job to confirm that the automation works as expected.
• If the automation doesn't work as expected, create the release environment 16.10 again manually by:
  • Revert https://gitlab.com/gitlab-com/gl-infra/release-environments/-/merge_requests/184.
  • Rerun the release-environments-deploy job of the pipeline in gitlab-org/gitlab (see an example here https://gitlab.com/gitlab-org/gitlab/-/pipelines/1219077783) to trigger a new pipeline in the release-environment.

Testing on 16.11.0

We can test the automation when releasing 16.11.0. Since this is a major release, every step in the release environment creation should happen automatically without human intervention. To make sure everything goes well, some checkpoints:

• Make sure the release_environment is activated on the OPS instance.
• The RC tag day is Tuesday, Apr 16, so set the Due Date of this issue to that day.
• Ask the RMs to inform when they tag the new release.
• Follow the release:tag job to confirm that the automation works as expected.
• If the automation doesn't work as expected, theoretically, we still have one more chance to fix and test it again, on the Tag Day Wednesday, Apr 17.

Exit Criteria

• Follow the step-by-step above
• Confirm that the automation works or not
  • If not, create follow up issue(s) to fix:
    •   #20163 (closed)