Wrap deployment notifications in a Slack thread
After #1241 (closed), a slack notification is sent to #announcements if a deployment is in progress and if there's an active incident or if there's something wrong with the health status checks. Since we often deploy with an incident active while we wait for the fix, the notifications can still get too noisy.
Proposal
Use Slack threads for the deployment notifications, this way the release managers can be notified just once and upcoming messages can be appended to the thread.
To do
-
Wrap up implementation gitlab-org/release-tools!1312 (merged) -
Test behavior -
Change the channel to #f_upcoming_release
- gitlab-org/release-tools!1334 (merged)
Test
Before testing, we need to:
-
Add variables in https://ops.gitlab.net/gitlab-org/release/tools: SLACK_TOKEN
-
Enable deployment_check
feature flag - https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/185/edit - Testing completed on #1432 (comment 472549089)
Deployment log
14/Dec
- Initial deployer job was wrong
- Slack class message was not sending parameters correctly
15/Dec
- Message to Slack was successfully sent - https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/2612896
- From observation, notifications always start with
gprd-migration 50%
, so this one should be considered by release-tools to be the initial deployer job. - We need to find a better way to share the Slack message ID so subsequent messages can be threaded
- On gitlab-org/release-tools!1329 (merged), we switch the strategy to use cache instead of artifacts
17/Dec
- It was noticed that Slack API requires
thread_ts
instead ofthread_id
- Thread messages are finally working
🎉
-
As a quick fix, it was decided to remove the noisiness from Slack messages, meaning:
- The initial message should contain the Slack foreword (active incidents and healthy health status)
- Threaded messages should only contain the deployment step information.
- Implemented on gitlab-org/release-tools!1332 (merged)
-
Initial deployer job strategy seems to be unreliable. An issue was opened to discuss alternatives #1432 (closed)
- We discovered the unreliability wasn't on the release-tools side but rather on the deployer side.
23/Dec
- Testing was successfully completed on #1432 (closed), you can see some examples on #1432 (comment 472549089)
How to disable?
- Disable
deployment_check
feature flag - https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/185/edit
Follow ups
- Ping Release Managers on the initial message - #1443
Edited by Mayra Cabrera