Skip to content

[Feature flag] Enable free_user_cap_over_limit_email

Summary

This issue is to roll out the feature on production, that is currently behind the free_user_cap_over_limit_email feature flag.

Owners

  • Most appropriate Slack channel to reach out to: #g_acquisition
  • Best individual to reach out to: @gdoud

Expectations

What are we expecting to happen?

Emails to be sent to owners of top level groups where adding a member to an invited group affects the group/project it was invited to in a way that causes it to go over the free user limit.

What can go wrong and how would we detect it?

logs indicate saturation due to influx of Namespaces::FreeUserCap::GroupOverLimitNotificationWorker flooding the sidekiq queue

Verification steps

Projects with invited groups

  1. Create a new top-level private group(A).
  2. Create a new project under group A project(P).
  3. Create a public(for ease of example so limit isn't applied) new top-level group(B).
  4. Go to the project A's member page and invite group B.
  5. Confirm on page reload no free user cap alerts exist(yet) and members are under 5.
  6. Go to the members page of group B and invite 6 users by username.
  7. Go to the members page of group A and confirm over limit alert message appears(not added by this MR).
  8. Verify email is received.
  9. Engineering: Verify logging events added in !138076 (merged) are seen.

Groups with invited groups

  1. Create a new top-level private group(A).
  2. Create a public(for ease of example so limit isn't applied) new top-level group(B).
  3. Go to the first top-level private group's member page and invite group B.
  4. Confirm on page reload no free user cap alerts exist(yet) and members are under 5.
  5. Go to the members page of group B and invite 6 users by username.
  6. Go to the members page of group A and confirm over limit alert message appears(not added by this MR).
  7. Verify email is received.
  8. Engineering: Verify logging events added in !138076 (merged) are seen.

Verification of both use cases completed as of 2024-01-23 ✅

Rollout Steps

Note: Please make sure to run the chatops commands in the Slack channel that gets impacted by the command.

Rollout on non-production environments

  • Verify the MR with the feature flag is merged to master and have been deployed to non-production environments with /chatops run auto_deploy status <merge-commit-of-your-feature>
  • Deploy the feature flag at a percentage (recommended percentage: 50%) with /chatops run feature set free_user_cap_over_limit_email <rollout-percentage> --actors --dev --staging --staging-ref
  • Monitor that the error rates did not increase (repeat with a different percentage as necessary).
  • Enable the feature globally on non-production environments with /chatops run feature set <feature-flag-name> true --dev --staging --staging-ref
  • Verify that the feature works as expected. The best environment to validate the feature in is staging-canary as this is the first environment deployed to. Make sure you are configured to use canary.
  • If the feature flag causes end-to-end tests to fail, disable the feature flag on staging to avoid blocking deployments.

For assistance with end-to-end test failures, please reach out via the #quality Slack channel. Note that end-to-end test failures on staging-ref don't block deployments.

Specific rollout on production

For visibility, all /chatops commands that target production should be executed in the #production Slack channel and cross-posted (with the command results) to the responsible team's Slack channel.

  • Ensure that the feature MRs have been deployed to both production and canary with /chatops run auto_deploy status https://gitlab.com/gitlab-org/gitlab/-/merge_requests/138076
  • Depending on the type of actor you are using, pick one of these options:
    • For project-actor: /chatops run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss,gitlab-com/www-gitlab-com free_user_cap_over_limit_email true
    • For group-actor: /chatops run feature set --group=gitlab-org,gitlab-com free_user_cap_over_limit_email true
    • For user-actor: /chatops run feature set --user=<your-username> free_user_cap_over_limit_email true
  • Verify that the feature works for the specific actors.

Preparation before global rollout

  • Set a milestone to this rollout issue to signal for enabling and removing the feature flag when it is stable.
  • Check if the feature flag change needs to be accompanied with a change management issue. Cross link the issue here if it does.
  • Ensure that you or a representative in development can be available for at least 2 hours after feature flag updates in production. If a different developer will be covering, or an exception is needed, please inform the oncall SRE by using the @sre-oncall Slack alias.
  • Ensure that documentation has been updated.
  • Leave a comment on the feature issue announcing estimated time when this feature flag will be enabled on GitLab.com.
  • Ensure that any breaking changes have been announced following the release post process to ensure GitLab customers are aware.
  • Notify the #support_gitlab-com Slack channel and your team channel (more guidance when this is necessary in the dev docs).
  • Ensure that the feature flag rollout plan is reviewed by another developer familiar with the domain.

Global rollout on production

For visibility, all /chatops commands that target production should be executed in the #production Slack channel and cross-posted (with the command results) to the responsible team's Slack channel (#g_TEAM_NAME).

  • Incrementally roll out the feature on production environment.
    • Between every step wait for at least 15 minutes and monitor the appropriate graphs on https://dashboards.gitlab.net.
    • Perform actor-based rollout:
    • 10% - Day 0: /chatops run feature set free_user_cap_over_limit_email 10 --actors
    • 25% - Day 1: /chatops run feature set free_user_cap_over_limit_email 25 --actors
    • 50% - Day 2: /chatops run feature set free_user_cap_over_limit_email 50 --actors
    • 75% - Day 3: /chatops run feature set free_user_cap_over_limit_email 75 --actors
    • 100% - Day 4: /chatops run feature set free_user_cap_over_limit_email true
  • Observe appropriate graphs on https://dashboards.gitlab.net and verify that services are not affected.
  • Leave a comment on the feature issue announcing that the feature has been globally enabled.
  • Wait for at least one day for the verification term.

Release the feature

After the feature has been deemed stable, the clean up should be done as soon as possible to permanently enable the feature and reduce complexity in the codebase.

You can either create a follow-up issue for Feature Flag Cleanup or use the checklist below in this same issue.

Feature Flag Clean-up issue: https://gitlab.com/gitlab-org/gitlab/-/issues/440214+

Rollback Steps

  • This feature can be disabled by running the following Chatops command:
/chatops run feature set free_user_cap_over_limit_email false
Edited by Kamil Niechajewicz