GitLab Next

After #1241 (closed), a slack notification is sent to #announcements if a deployment is in progress and if there's an active incident or if there's something wrong with the health status checks. Since we often deploy with an incident active while we wait for the fix, the notifications can still get too noisy.

Proposal

Use Slack threads for the deployment notifications, this way the release managers can be notified just once and upcoming messages can be appended to the thread.

To do

• Wrap up implementation gitlab-org/release-tools!1312 (merged)
• Test behavior
• Change the channel to #f_upcoming_release - gitlab-org/release-tools!1334 (merged)

Test

Before testing, we need to:

• Add variables in https://ops.gitlab.net/gitlab-org/release/tools: SLACK_TOKEN
  • Taken from https://ops.gitlab.net/gitlab-com/chatops/-/settings/ci_cd
• Enable deployment_check feature flag - https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/185/edit
• Testing completed on #1432 (comment 472549089)

Deployment log

14/Dec

• Initial deployer job was wrong
  • Noted on https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/2601733
  • Fixed by gitlab-org/release-tools!1323 (merged)
• Slack class message was not sending parameters correctly
  • Noted on https://sentry.gitlab.net/gitlab/release-tools/issues/2416145/?referrer=slack
  • Fixed by gitlab-org/release-tools!1326 (merged)

15/Dec

• Message to Slack was successfully sent - https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/2612896

• From observation, notifications always start with gprd-migration 50%, so this one should be considered by release-tools to be the initial deployer job.
• We need to find a better way to share the Slack message ID so subsequent messages can be threaded
  • On gitlab-org/release-tools!1329 (merged), we switch the strategy to use cache instead of artifacts

17/Dec

• It was noticed that Slack API requires thread_ts instead of thread_id
  • Fixed by gitlab-org/release-tools!1331 (merged)
• Thread messages are finally working 🎉

• As a quick fix, it was decided to remove the noisiness from Slack messages, meaning:

  • The initial message should contain the Slack foreword (active incidents and healthy health status)
  • Threaded messages should only contain the deployment step information.
  • Implemented on gitlab-org/release-tools!1332 (merged)

• Initial deployer job strategy seems to be unreliable. An issue was opened to discuss alternatives #1432 (closed)

  • We discovered the unreliability wasn't on the release-tools side but rather on the deployer side.

23/Dec

• Testing was successfully completed on #1432 (closed), you can see some examples on #1432 (comment 472549089)

How to disable?

• Disable deployment_check feature flag - https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/185/edit

Follow ups

• Ping Release Managers on the initial message - #1443