Add Slack notifications to the release-tools coordinated pipeline
Problem
When a deployment to any of our environments starts or finishes, a slack notification is sent to the #announcements
channel. Several examples of this notification can be found on the announcements
slack channel.
Slack notification logic is centralized in two places: Notification is sent using a python script on the deploy-tooling repository, then it's referenced by the deployer pipeline.
Proposal
To reduce the complexity of the Slack notification setup and as part of the Single Pipeline::Phase 2, let's add the Slack notification to the coordinated pipeline:
- Coordinated pipeline should send a notification when a deployment starts, ends, or fails.
- If a notification is sent from the coordinated pipeline, the deployer pipeline notification should be skipped.
Note that we can't get rid of the deployer notification due to the different ways a deployment can be triggered:
- Standard way: Using coordinated pipelines through release-tools - This one will use the new notifications
- Alternative way: Using a chatops command - This one will use the deployer notifications - This option is normally used if:
- Production is blocked due to incidents - Example (Internal only)
- If we want to speed up the deployment of a package
- If we want to deploy a specific package into one of our environments (rollbacks)
Follow-up issue to remove notifications from deployer #2070
Implementation details
Moving this notification implies:
For context, deployments to our environments are performed using bridge-jobs, meaning a downstream pipeline is triggered on the deployer repo to perform a deployment - example of the bridge job for staging
-
Fetch the downstream pipelines through the bridges API, build a Slack notification based on its info gitlab-org/release-tools!1538 (merged) -
Modify .gitlab-ci.yml
to include the new stages and jobs gitlab-org/release-tools!1549 (merged) -
Display additional elements in the slack messages - gitlab-org/release-tools!1560 (merged) -
Test the notifications using a different channel #deployment_notification_tests
#announcements
Slack channel
Transition to the Once the testing has been deemed stable, we could start using the new notifications in the auto-deploy process
-
Modify the deployer pipeline to skip notifications if indicated. - https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/412 -
Update the code to use #announcements
channel gitlab-org/release-tools!1573 (merged) -
Have both MRs approved
Oct 6th
-
Before the creation of the auto-deploy branch scheduled at 15 UTC, merge both merge requests -
Keep an eye on the auto-deploy branch created at that schedule, and make sure the notifications are correct, and sent properly to the #announcements
channel
Result: Slack notifications are sent to the #announcements
channel through the coordinated pipeline. No error has been reported so far.
What to do if something goes wrong?
- Revert gitlab-org/release-tools!1573 (merged)
- Trigger a new coordinated pipeline through the Pipelines page
Development log
- August 31st - Implementation changes started with an MR that introduces the classes to send slack notifications was opened gitlab-org/release-tools!1538 (merged)
-
September 7th
- Modifications to our CI configuration were started on gitlab-org/release-tools!1549 (merged). gitlab-org/release-tools!1538 (merged)
- gitlab-org/release-tools!1538 (merged) is in review
- Two MRs were created based on the review:
- Standardizes deploy canary job - gitlab-org/release-tools!1551 (merged)
- Add an alias for rspec-doubles - gitlab-org/release-tools!1552 (merged)
-
September 13th
- gitlab-org/release-tools!1538 (merged) was merged
- gitlab-org/release-tools!1549 (merged) is in review
- September 14th - An MR implementing additional elements (timing, wall duration, sentry link) was opened gitlab-org/release-tools!1560 (merged)
- September 15th - gitlab-org/release-tools!1560 (merged) was merged
-
September 20th - gitlab-org/release-tools!1549 (merged) was merged.
- A follow-up was created to split notifications based on the deployment status (
failed
orsuccess
) #2034 (closed)
- A follow-up was created to split notifications based on the deployment status (
-
September 21st
- A bug was found: A deployment started after a failure gitlab-org/release-tools!1570 (merged)
- Implementation of splitting notifications based on their status was opened gitlab-org/release-tools!1568 (merged)
- September 23rd - gitlab-org/release-tools!1568 (merged) was submitted to review
- September 27th - gitlab-org/release-tools!1568 (merged) was merged.
-
September 29th
- We noticed that parent pipelines are treated differently if a downstream pipeline is
canceled
#1489 (comment 688377332), this is a bug in the application #1489 (comment 689306743). Other than that, testing has been positive so far (see #1489 (comment 689562206)), so we're ready to update the deployer pipeline for the transition.
- We noticed that parent pipelines are treated differently if a downstream pipeline is
-
October 5
- Final changes for release-tools gitlab-org/release-tools!1573 (merged) and deployer https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/412/diffs have been submitted and approved. I plan to merge those tomorrow and start the final testing.
-
October 6-7
- Slack notifications are sent to the
#announcements
channel through the coordinated pipeline. No error has been reported so far.
- Slack notifications are sent to the
Follow ups
- Remove feature flags #2071 (closed)
- Remove the slack notifications from deployer #2070