Skip to content
GitLab
Next
    • GitLab: the DevOps platform
    • Explore GitLab
    • Install GitLab
    • How GitLab compares
    • Get started
    • GitLab docs
    • GitLab Learn
  • Pricing
  • Talk to an expert
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
    Projects Groups Snippets
  • Sign up now
  • Login
  • Sign in / Register
  • delivery delivery
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 552
    • Issues 552
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
  • Merge requests 0
    • Merge requests 0
  • Deployments
    • Deployments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • Code review
    • Insights
    • Issue
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Commits
  • Issue Boards
Collapse sidebar
  • GitLab.comGitLab.com
  • GitLab Infrastructure TeamGitLab Infrastructure Team
  • deliverydelivery
  • Issues
  • #1241
Closed
Open
Issue created Sep 29, 2020 by Alessio Caiazza@nolith🚀Maintainer12 of 12 checklist items completed12/12 checklist items

Keep track of deployment advancement and production status

This issue is about extending release-tools to keep track of the ongoing deployment.

With #1094 (closed) we will trigger production checks at specific checkpoints during a production deployment. (TL;DR: implementation recap in #1241 (comment 434063792))

We need a proper way to handle those checks in release-tools.

At the moment, the baking time message is the closest thing to what we need

image

Proposed implementation

  • Modify ReleaseTools::Tasks::AutoDeploy::CheckProduction to accept some additional ENV parameters

    • DEPLOYMENT_CHECK: will be set to "true" to trigger the deployment advanced status
    • DEPLOYMENT_STEP: a string describing in which point of the deployment we are at the moment (this is url-encoded)
    • DEPLOYER_JOB_URL: the URL of the deployer job triggering the check
  • If DEPLOYMENT_CHECK==true a new ReleaseTools::Promotion::Manager#deployment_check method will be invoked

  • The baking time foreword is replaced with a specific message

    :an_emoji_that_makes_sense: Production deployment reached ${DEPLOYMENT_STEP}
    Package version: `the version`
    Pipeline: ${DEPLOYER_JOB_URL}
  • Rework the summary line with something like The deployment can continue/Please @release-manager investigate if it's worth halting the deployment

  • checks are ok in the current format

To do

  • Wrap up implementation gitlab-org/release-tools!1254 (merged)
  • Enabled the feature flag and notify teamDelivery
    • deployment_check FF for release-tools - https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/185/edit
    • Add TRIGGER_INTERMEDIATE_PRODUCTION_CHECKS ENV variable to the deployer project - https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/settings/ci_cd
  • Fix DEPLOYER_JOB_URL and DEPLOYMENT_STEP variables
    • https://ops.gitlab.net/gitlab-com/gl-infra/deploy-tooling/-/merge_requests/323
    • gitlab-org/release-tools!1285 (merged)
  • Complete #1346 (closed)
    • Only send notifications if the deployment is in progress and the production checks are failing
    • Move the notification to the #announcement Slack channel
  • Limit the notification to be sent only in deployments to prod - gitlab-org/release-tools!1304 (closed) / https://ops.gitlab.net/gitlab-com/gl-infra/deploy-tooling/-/merge_requests/324
  • Document TRIGGER_INTERMEDIATE_PRODUCTION_CHECKS env variable https://gitlab.com/gitlab-org/release/docs/-/blob/master/runbooks/variables.md#deployer - gitlab-org/release/docs!304 (merged)
  • Wrap the notifications in a single Slack thread - #1372 (closed)
  • Open follow ups:
    • To remove the feature flag - #1373 (closed)
    • To restore the release managers mention #1443

Development log

  • Initial implementation was done on gitlab-org/release-tools!1254 (merged)
  • Messages were considered noisy by the team because of it:
    • Messages are only sent if a deployment is on-going and if there's an active incident (or a problem with health status) #1346 (closed)
    • Messages should be wrapped up in Slack threads #1372 (closed)
  • Testing was successfully completed on #1432 (closed), you can see some examples on #1432 (comment 472549089)
  • Notifications were moved to f_upcoming_release channel on gitlab-org/release-tools!1334 (merged)

How to turn it off?

  • Disable the feature flag on release tools https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags/185/edit
  • Remove TRIGGER_INTERMEDIATE_PRODUCTION_CHECKS from deployer - https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/settings/ci_cd
Edited Dec 23, 2020 by Mayra Cabrera
Assignee
Assign to
Time tracking