[Feature flag] Rollout of `ssh_key_expiration_email_notification`

Feature

The SSH key expiration notification email feature (both the "already expired" and "expiring soon" emails) is behind the :ssh_key_expiration_email_notification feature flag. The feature flag needs to be rolled out.

Note: These emails are scheduled to be sent at 1am, presumably that is UTC.

Owners

• Team: groupcompliance
• Most appropriate slack channel to reach out to: #g_manage_compliance
• Best individual to reach out to: @asubramanian1
• PM:

Stakeholders

The Rollout Plan

• Partial Rollout on GitLab.com with beta groups
• Rollout on GitLab.com for a certain period (How long)
• Percentage Rollout on GitLab.com - XX% If it is possible to perform an incremental rollout, this should be preferred. Proposed increments are: 10%, 50%, 100%. Proposed minimum time between increments is 15 minutes.
• Rollout Feature for everyone as soon as it's ready

Beta Groups/Projects:

• gitlab-org/gitlab project
• gitlab-org/gitlab-com groups
• ...

Expectations

What are we expecting to happen?

What might happen if this goes wrong?

What can we monitor to detect problems with this?

Rollout Timeline

Initial Rollout

Preperation Phase

• Enable on staging (/chatops run feature set ssh_key_expiration_email_notification true --staging)

• Test on staging

• Ensure that documentation has been updated (More info)

• Coordinate a time to enable the flag with the SRE oncall and release managers

  • In #production by pinging @sre-oncall
  • In #g_delivery by pinging @release-managers

• Announce on the issue an estimated time this will be enabled on GitLab.com

Partial Rollout Phase

• Enable on GitLab.com for individual groups/projects listed above and verify behaviour (/chatops run feature set --project=gitlab-org/gitlab ssh_key_expiration_email_notification true)

• Verify behaviour (See Beta Groups) and add details with screenshots as a comment on this issue

Global Availability (More Info) (Please Note that Beta, Alpha and General Availability (GA) are handled on a product level and not the feature-flag)

• Coordinate a time to enable the flag with #production and #g_delivery on slack.

• Announce on the issue an estimated time this will be enabled on GitLab.com

• Make the feature flag enabled by default i.e. Change default_enabled to true in !61812 (merged)

• Enable on GitLab.com by running chatops command in #production (/chatops run feature set ssh_key_expiration_email_notification 10)

• Enable on GitLab.com by running chatops command in #production (/chatops run feature set ssh_key_expiration_email_notification true)

• Announce on the issue that the flag has been enabled

• Cross post chatops slack command to #support_gitlab-com (more guidance when this is necessary in the dev docs) and in your team channel

Rollback Steps

• This feature can be disabled by running the following Chatops command:

/chatops run feature set ssh_key_expiration_email_notification false

added feature flag label

changed the description

changed milestone to %13.11

assigned to @asubramanian1

added groupcompliance sectiondev labels

marked the checklist item Enable on staging (/chatops run feature set feature_name true --staging) as completed

marked the checklist item Ensure that documentation has been updated (More info) as completed

marked the checklist item Rollout on GitLab.com for a certain period (How long) as completed

Setting label(s) devopsmanage based on groupcompliance.

added devopsmanage label

changed milestone to %13.12

set weight to 1

added workflowready for development label

unassigned @asubramanian1

added [deprecated] Accepting merge requests label

marked the checklist item Test on staging as completed

@tancnle @jiaan it looks like you both reviewed some of this code, so I'm hoping you can help me with the rollout. This flag enabled in Staging 2021-04-08, and I'm not aware of any complaints. Any objections to starting the rollout in production?

Note: The only piece of this that hasn't been merged already is the omnibus cron job MR.

cc @mwoolf

@djensen I think we can progressively rolling it out on prod. I believe the omnibus cron job MR is only required for self-managed customers. What do you think about pushing for Monday 17/05?

@tancnle @mwoolf yes, let's push hard for Monday the 17th, because this is an important feature to release prior to the breaking change. I just enabled at 10% production (although 10% may be ineffectual) and created an MR to enable by default. Please carry this work forward and consider it your top priority. We should make every effort to get this into %13.12 so users with expired SSH keys have plenty of warning prior those keys no longer functioning.

changed the description

assigned to @tancnle