Internal releases: Test the internal release process
📖 Context
To remediate GiLab single-tenant instances within remediation SLAs, a new GitLab internal release strategy is being designed and introduced on &1201 (closed).
GitLab releases have been public by default, introducing a new private release strategy is an uncharted territory that should be thoroughly tested before using it as a remediation tool on high-severity incidents.
This issue aims to test the end-to-end process of an internal release to guarantee the standard workflow works as expected. The testing is split into five phases:
- Phase 1 - Internal release initial process
- Phase 2 - Ability to generate packages
- Phase 3 - Verification of internal package
- Phase 4 - Final steps of the internal release
📓 General Requirements
- Tests should not be performed during patch release weeks.
- Coordination with release managers is required
- Testing start and ending should be notified to
#g_distribution
and#g_dedicated_team
Slack channels - Enable the
internal_release
feature flag https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags
✅ Phase 1: Internal release initial process
Click to expand
Setup
-
Backport Omnibus and CNG changes to 17.6 and 17.5 #20845 (closed) -
To prevent spamming release managers remove assignees from the internal release issue gitlab-org/release-tools!3845 (merged) -
Notify release managers about the test -
Enable the internal_release
feature flag https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags - [-] Set
SLACK_TEST
on https://ops.gitlab.net/gitlab-org/release/tools/-/settings/ci_cd
Start the internal release process
-
Kick off the internal release process: /chatops run release internal
-
Confirm a release task issue is automatically created. https://gitlab.com/gitlab-org/release/tasks/-/issues/16724 -
The release task issue should -
Use the internal release versions: 17.7 and 17.6 -
Be assigned to active release managers (temporarily to mayra-cabrera
) -
Include the required steps to perform an internal release -
Have the internal release pipeline created https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/4139583
-
Internal release with a specific iteration
-
Kick off the internal release process /chatops run release internal --iteration=\<internal_number\>
-
Confirm a release task issue is automatically created considering the iteration number https://gitlab.com/gitlab-org/release/tasks/-/issues/16725
Initial announcements.
-
Trigger the internal_release_prepare:start
process -
Verify a Slack alert has been sent to #f_upcoming_release_channel
-
Confirm the internal_release_prepare:notify_dedicated
job is executed (this job is no-op)
Observations
Details: #20727 (comment 2321316703)
✅ Phase 2: Creation of an internal release package
First run
Setup:
-
Prepare a merge request targeting the 17.7 canonical stable branch gitlab-org/gitlab!179735 (merged) -
Prepare a merge request targeting the 17.6 canonical stable branch gitlab-org/gitlab!179736 (merged) -
Generate a new internal release pipeline with SLACK_TEST
https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/4143174 -
Link the internal release pipeline to https://gitlab.com/gitlab-org/release/tasks/-/issues/16724 -
Enable the internal_release
feature flag https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags -
Merge canonical merge requests: gitlab-org/gitlab!179735 (merged), gitlab-org/gitlab!179736 (merged) -
Notify release managers. -
Notify Distribution -
Notify Dedicated
Job execution
-
Trigger the internal_release_release:start
job -
Verify a Slack alert has been sent to #release_tools_tests
-
Verify the internal_release:trigger_service
is successfully executed
Package creation
17.7
-
Verify a commit is updated on the GitLab stable branch https://gitlab.com/gitlab-org/security/gitlab/-/commits/17-7-stable-ee -
Verify an Omnibus pipeline is triggered: -
Verify the CNG pipelines are triggered: -
Regular pieline: -
UBI pipeline: -
FIPS pipeline:
-
17.6
-
Verify a commit is updated on the GitLab stable branch https://gitlab.com/gitlab-org/security/gitlab/-/commits/17-6-stable-ee -
Verify an Omnibus pipeline is triggered: -
Verify the CNG pipelines are triggered: -
Regular pieline: -
UBI pipeline: -
FIPS pipeline:
-
Metadata
17.7
-
Verify a metadata entry is successfully created on ops instance https://ops.gitlab.net/gitlab-org/release/metadata -
Metadata entry: Verify it uses the last stable branch sha for every GitLab component: -
GitLab https://gitlab.com/gitlab-org/security/gitlab/-/commits/17-7-stable-ee -
Omnibus https://gitlab.com/gitlab-org/security/omnibus-gitlab/-/commits/17-7-stable -
CNG https://gitlab.com/gitlab-org/security/charts/components/images/-/tree/17-7-stable -
Gitaly https://gitlab.com/gitlab-org/security/gitaly/-/commits/17-7-stable -
GitLab Pages https://gitlab.com/gitlab-org/security/gitlab-pages/-/commits/17-7-stable
-
17.6
-
Verify a metadata entry is successfully created on ops instance https://ops.gitlab.net/gitlab-org/release/metadata -
Metadata entry: Verify it uses the last stable branch sha for every GitLab component: -
GitLab https://gitlab.com/gitlab-org/security/gitlab/-/commits/17-6-stable-ee -
Omnibus https://gitlab.com/gitlab-org/security/omnibus-gitlab/-/commits/17-6-stable -
CNG https://gitlab.com/gitlab-org/security/charts/components/images/-/tree/17-6-stable -
Gitaly https://gitlab.com/gitlab-org/security/gitaly/-/commits/17-6-stable -
GitLab Pages https://gitlab.com/gitlab-org/security/gitlab-pages/-/commits/17-6-stable
-
Slack announcement
-
Verify a Slack alert has been sent to #f_upcoming_release_channel
with the job status
Second run
Setup:
-
Prepare a merge request targeting the 17.8 canonical stable branch gitlab-org/gitlab!182441 (merged) -
Prepare a merge request targeting the 17.7 canonical stable branch gitlab-org/gitlab!182443 (merged) -
Generate a new internal release pipeline with notifications off: https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/4243119 -
SLACK_TEST
=TRUE
-
INTERNAL_RELEASE_PIPELINE
=TRUE
-
-
Generate a new release/task issue https://gitlab.com/gitlab-org/release/tasks/-/issues/17690 -
Link the internal release pipeline to the release/task issue -
Enable the internal_release
feature flag https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags -
Merge canonical merge requests:
Kick off the internal release process
-
Trigger the internal_release_prepare:start
process -
Confirm the internal_release_prepare:check_component_branch_pipeline_status
is successfully executed. https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/17444110- This job will check the status of the 17.8 and 17.7 branches
-
Confirm the internal_release_prepare:notify_dedicated
is successfully executed. This job is a no-op
Job execution
-
Trigger the internal_release_release:start
job -
Verify a Slack alert has been sent to #release_tools_tests
-
Verify the internal_release_release:create_dynamic_pipeline
is successfully executed. -
Verify a downstream pipeline is generated with a job for each version. https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/4243154 -
Verify the internal_release:trigger_service
is successfully executed
Package creation
17.8
-
Verify a commit is updated on the GitLab stable branch https://gitlab.com/gitlab-org/security/gitlab/-/commits/17-8-stable-ee -
Verify an Omnibus pipeline is triggered: https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipelines/369218 -
Verify CNG pipelines are triggered:
17.7
-
Verify a commit is updated on the GitLab stable branch https://gitlab.com/gitlab-org/security/gitlab/-/commits/17-7-stable-ee -
Verify an Omnibus pipeline is triggered: https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipelines/369205 -
Verify the CNG pipelines are triggered:
Metadata
17.8
-
Verify a metadata entry is successfully created on ops instance https://ops.gitlab.net/gitlab-org/release/metadata/-/commit/38b88acc5722ed370a79efdd571d63708f9bf840 -
Metadata entry: Verify it uses the last stable branch sha for every GitLab component: -
GitLab https://gitlab.com/gitlab-org/security/gitlab/-/commits/17-8-stable-ee -
Omnibus https://gitlab.com/gitlab-org/security/omnibus-gitlab/-/commits/17-8-stable -
CNG https://gitlab.com/gitlab-org/security/charts/components/images/-/tree/17-8-stable -
Gitaly https://gitlab.com/gitlab-org/security/gitaly/-/commits/17-8-stable -
GitLab Pages https://gitlab.com/gitlab-org/security/gitlab-pages/-/commits/17-8-stable
-
17.7
-
Verify a metadata entry is successfully created on ops instance https://ops.gitlab.net/gitlab-org/release/metadata/-/commit/bbe48c279f739ebfa76bf4afebbf971826b5706d -
Metadata entry: Verify it uses the last stable branch sha for every GitLab component: -
GitLab https://gitlab.com/gitlab-org/security/gitlab/-/commits/17-7-stable-ee -
Omnibus https://gitlab.com/gitlab-org/security/omnibus-gitlab/-/commits/17-7-stable -
CNG https://gitlab.com/gitlab-org/security/charts/components/images/-/tree/17-7-stable -
Gitaly https://gitlab.com/gitlab-org/security/gitaly/-/commits/17-7-stable -
GitLab Pages https://gitlab.com/gitlab-org/security/gitlab-pages/-/commits/17-7-stable
-
Slack announcement
-
Verify a Slack alert has been sent to #f_upcoming_release_channel
with the job status
Observations:
- First run: #20727 (comment 2326315412)
- Second run: #20727 (comment 2373612695)
✅ Phase 3: Availability of the package
Click to expand
Job execution
-
After 150 minutes the internal package was tagged, confirm the internal_release_verify:start
automatically starts -
Verify a Slack alert has been sent to #f_upcoming_release_channel
-
Verify the internal_release_verify:check_package_build
job has successfully executed
Internal packages verification
- 17.8
-
Check the package is available on the pre-release channel https://packages.gitlab.com/gitlab/pre-release/ -
Check the package is dev registry (https://dev.gitlab.org/gitlab/charts/components/images/container_registry/393?orderBy=NAME&sort=asc&search%5B%5D=17.8.4-internal0
- 17.7
-
Check the package is available on the pre-release channel https://packages.gitlab.com/gitlab/pre-release/ -
Check the package is dev registry https://dev.gitlab.org/gitlab/charts/components/images/container_registry/393?orderBy=NAME&sort=asc&search%5B%5D=17.7.6-internal0
Slack announcement
-
Check the internal_release_verify:check_package_build
notify to Slack its completion
Observations:
✅ Phase 4: Internal release final steps
Click to expand
-
Trigger the internal_release:finalize:start
job on the internal release pipeline -
Verify a Slack alert has been sent to #f_upcoming_release_channel
about the start of this phase -
Release manager notification -
Verify a Slack notification was sent to #f_upcoming_release_channel
notifying release managers the package is available -
Verify a comment was added to the internal release task
-
-
Dedicated: Verify the internal_release_finalize:notify_dedicated
is automatically executed (this job is no-op)
Observations
Clean up
-
Restore gitlab-org/release-tools!3845 (merged) => gitlab-org/release-tools!3952 (merged) -
Restore gitlab-org/release-tools!3949 (merged) => gitlab-org/release-tools!3952 (merged) -
Disable the internal_release
feature flags -
Create follow-up issues -
Close https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/20651 if applicable
Follow-ups
- Internal release: Version file on GitLab repo i... (#20858 - closed)
- Internal releases: Merge-train on stable branch... (#20860 - closed)
- Internal release: Release job failed due to co... (#20859 - closed)
- Ensure the timeout of the `internal_release_rel... (#20927 - closed)
- Execute metadata independent of the internal re... (#20928 - closed)
- CNG verification on internal release template (#20933 - closed)
- Add a Slack status notification to the internal... (#20934 - closed)
- https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/20906+
- Test the merge train process on internal releases (#20940 - closed)
Edited by Mayra Cabrera