Internal release tools testing for the maintenance policy extension

The work towards extending the maintenance policy is in progress, however, modifications for the release toolings are already in place. The purpose of this issue is to perform internal testing to ensure release managers have the required tooling to perform patch releases for three versions.

This internal testing is a 'dry-run' in terms that no patch release version should be tagged or published.

Exit criteria

• Guidance is provided when merging merge requests into stable branches
  • type::bug label is enforced
  • package-and-test pipeline execution is encouraged
  • Merge requests outside the policy throw a warning
• Tooling
  • Patch release pressure reflects the state of the MRs merged into stable branches
  • Patch release task issue is generated
  • Blog posts accounts for bug fixes merged into stable branch
  • The steps in the issue template account for three versions

Requirements

Merge requests targeting 15-9-stable-ee should be prepared.

• For the merge requests, identify typos throughout the app:

List

• Doc changes:
  • doc/development/rake_tasks.md:261:66: seperated should be separated
  • doc/development/pages/index.md:227:8: specfic should be specific
• Spec changes:
  • spec/models/error_tracking/project_error_tracking_setting_spec.rb:418:61: appers should be appears
  • spec/models/error_tracking/project_error_tracking_setting_spec.rb:305:46: relase should be release
  • spec/models/container_registry/event_spec.rb:137:20: respository should be repository
• App changes:
  • app/models/container_registry/event.rb:59:15: existance should be existence
  • app/models/ci/job_artifact.rb:373:13: overriden should be overridden
  • app/models/ci/job_artifact.rb:377:13: overriden should be overridden

• Prepare merge requests targeting 15-9-stable-ee. Merge requests should contain simple changes, for example: documentation updates, fix for flaky failures, or small backend and frontend changes.
  • Doc changes - gitlab-org/gitlab!112612 (merged)
  • Spec changes - gitlab-org/gitlab!112613 (merged)
  • App changes - gitlab-org/gitlab!112615 (closed)
• Create backport merge request to the respective stable branch.

Merge request process

✅ With a properly set merge request

• Tested on 2023-02-23
• MR: gitlab-org/gitlab!112615 (closed)

Tests:

• Ensure the merge request has the type::bug label
• Verify Danger should not fail https://gitlab.com/gitlab-org/gitlab/-/jobs/3824560803
• Ensure the package-and-test pipeline is automatically executed
• Ensure the Reviewer Roulette is displayed as part of Danger
• Ensure the package-and-test execution is encouraged

Package-and-qa message (gitlab-org/gitlab!112615 (comment 1287297821))

Observations

Small details were found on #2839 (comment 1287312560). Fixes were created:

• Skip danger check on documentation MRs (gitlab-org/gitlab!112751 - merged)
• Remove pick into x.y message from Danger (gitlab-org/gitlab!112740 - merged)
• Split QA into a dedicated section (gitlab-org/gitlab!112742 - merged)
• Introduce a stable branch MR template (gitlab-org/gitlab!112736 - merged)

✅ With a merge request using the wrong labels

• Tested on 2023-02-23
• MR gitlab-org/gitlab!112613 (merged)

• Add the type::feature label and verify that Danger fails
• Add the pipeline:expedite label and verify that Danger fails

Danger job	Labels	Danger error messages
https://gitlab.com/gitlab-org/gitlab/-/jobs/3824942095
https://gitlab.com/gitlab-org/gitlab/-/jobs/3824992959

Package-and-test pipeline

✅ General

Tests

• Danger pings the author and states the package-and-test pipeline should be executed, and failures should be reviewed by the SET counterpart

Package-and-test	Danger job	Warning message
https://gitlab.com/gitlab-org/gitlab/-/pipelines/784674631	gitlab-org/gitlab!112615 (comment 1287297821)

Observations:

• The Danger message could be highlighted as an action that needs to be taken. gitlab-org/gitlab!112855 (merged)
• The message states "See the section below for more details. But this could be ambiguous since are many sections, the message should state what Danger section should be read. gitlab-org/gitlab!112855 (merged)

✅ When the package-and-test is not present

• Guidance about the requirement of package-and-test should be indicated by Danger.
• Danger should fail

Observations:

• The error message states the package-and-test needs to be manually triggered, this is not the case on stable branches because the pipeline is automatically executed. gitlab-org/gitlab!112855 (merged)

MR	Danger job	Danger error message
gitlab-org/gitlab!112743 (closed)	https://gitlab.com/gitlab-org/gitlab/-/jobs/3825789810

✅ Status: canceled

• Danger pings the author and states the package-and-test pipeline should be executed, and the SET counterpart should review failures
• Danger should fail

package-and-test	Danger job	Error	Message
https://gitlab.com/gitlab-org/gitlab/-/pipelines/795615508	https://gitlab.com/gitlab-org/gitlab/-/jobs/3872422308

Observations

• Danger didn't fail because it is assuming the bridge status rather than the downstream pipeline status gitlab-org/gitlab!113272 (diffs)

✅ Status: success

• Danger pings the author and states the package-and-test pipeline should be executed, and the SET counterpart should review failures

package-and-test	Danger job	Warning	Message
https://gitlab.com/gitlab-org/gitlab/-/pipelines/793261571	https://gitlab.com/gitlab-org/gitlab/-/jobs/3858303015	N/A

Observations:

• The message states 'manually triggered'. On stable branches, this message is automatically triggered so the message is inaccurate. It may also be a good idea to refer engineers to the Package-and-test section gitlab-org/gitlab!112855 (merged)

✅ Status failed

• Danger pings the author and states the package-and-test pipeline should be executed, and the SET counterpart should review failures

package-and-test	Danger job	Warning	Message
https://gitlab.com/gitlab-org/gitlab/-/pipelines/784674631	https://gitlab.com/gitlab-org/gitlab/-/jobs/3858303015

✅ Documentation and flaky merge requests

• Ensure the package-and-test pipeline is not encouraged on documentation MRs
• Ensure the package-and-test pipeline is not encouraged on merge requests with flaky::failure-test label

	Documentation MRs	MRs with flaky::failure-test
Danger message
Danger job	https://gitlab.com/gitlab-org/gitlab/-/jobs/3873200250	https://gitlab.com/gitlab-org/gitlab/-/jobs/3873226158
Merge request	gitlab-org/gitlab!112612 (merged)	gitlab-org/gitlab!112613 (merged)

Observations:

• Is it possible to skip the package-and-test execution for documentation changes?
  • Technically should be possible but this is a nice-to-have. Added to the nice to have section.
• The package-and-test pipeline was actually encouraged on both, merge requests with documentation changes and merge requests fixing flaky failures. Fixed by gitlab-org/gitlab!113431 (merged)

✅ Merge requests targeting versions outside the policy

• Ensure a Danger message is listed specifying the version is outside the policy.

MR (gitlab-org/gitlab!113436 (comment 1299441328))

Backport MRs:

• Skip danger check on documentation MRs (gitlab-org/gitlab!112751 - merged)
• Remove pick into x.y message from Danger (gitlab-org/gitlab!112740 - merged)
• Split QA into a dedicated section (gitlab-org/gitlab!112742 - merged)
• Introduce a stable branch MR template (gitlab-org/gitlab!112736 - merged)
• Highlight package-and-qa Danger message (gitlab-org/gitlab!112855 - merged)
• Fail danger if package-and-test pipeline is can... (gitlab-org/gitlab!113272 - merged)
• Skip package-and-test message on trivial MRs (gitlab-org/gitlab!113431 - merged)
• Prevent QA check on stable branches (gitlab-org/gitlab!113281 - merged)
• Skip Danger failure when package-and-test is be... (gitlab-org/gitlab!113564 - merged)

Backported on gitlab-org/gitlab!113549 (merged) and gitlab-org/gitlab!113871 (merged)

Test with the backports merged and backported into the stable branch #2839 (comment 1299525695)

✅ Path release pressure

• Merge the merge requests into the stable branch
• Ensure the patch release pressure is increased

Status	MR gitlab-org/gitlab!112612 (merged)	MR gitlab-org/gitlab!112613 (merged)	Patch release pressure (general)	Patch release pressure (by version)
✅	severity1	severity2
✅	severity1	severity3
✅	severity3	severity4
✅	N/A	N/A

✅ Tooling verification

• Execute /chatops run release prepare https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/9423218
• Ensure only the release task issue and the blog post are created
  • Release task issue: gitlab-org/release/tasks#5183 (closed)
  • Blog post: gitlab-com/www-gitlab-com!120963 (closed)
• Corroborate the blog post created contains the merge requests
• Follow the steps listed and ensure they make sense

Observations

• After creating the blog post, there is no easy way for this one to be updated if merge requests are merged after that. The workaround is to close the blog post and re-trigger the /chatops run release prepare command

✅ With a failure on a stable branch

• Ensure a message is posted on #releases
• Ensure an issue is opened in https://gitlab.com/gitlab-org/release/tasks

---

Nice to haves

• Skip the package-and-test pipeline on documentation MRs - gitlab-org/quality/quality-engineering/team-tasks#1691
• Skip the message suggesting a type:: label should be added. This is taken care of by Danger. #2894
• Checking stable branches for projects under Managed Versioning is super manual, it would be nice to have some automation for this (this will be a nice to have though) #2895
• Define a patch release threshold #2896 (closed)
• Automatically alert RM about S1/S2 unreleased merge requests. #2897 (closed)
• After creating the blog post, there is no easy way for this one to be updated if merge requests are merged after that. The workaround is to close the blog post and re-trigger the /chatops run release prepare command #2898 (closed)