Add workers to notify over limit owners

SAM FIGUEROArequested to merge
sf/feature/GL367449-add-workers-to-trigger-over-limit-mails into master

What does this MR do and why?

  • Part of #367549+

  • Add a worker to be triggered by cron(settings for the cron will come in a follow-up MR), to schedule another worker that sends an email notification to the group owner that's gone over the free user limit. Both observe if the dashboard_limit is activated, and are also guarded by the activation of the free_user_cap_over_user_limit_mails feature flag.

  • Add a finder to query the DB for groups that could potentially be over limit according to the free user criteria. Should make it easy to adapt the parameters if we pivot to another method for determining the cap.

Performance considerations

  • We've changed the job OverLimitNotificationWorker to implement the Limited::Capacity concern with a current max jobs of 5 and a batch size of 1. It will keep processing groups until it has reached processed all unprocessed groups that apply to the enforcement criteria (free, top-level, and private).
    • The constants MAX_RUNNING_JOBS & BATCH_LIMIT are used to scale the concurrency (up or down)
  • The jobs are guarded by a feature flag free_user_cap_over_user_limit_mails
    • The mailer job will silently exit unless the flag is set

over_limit? check

meta: Work on improving performance of the over_limit was completed in https://gitlab.com/gitlab-org/gitlab/-/issues/383264+ and has significantly reduced the query times for this check.

Potential Namespaces Effected

  • This Sisense Board reads roughly 100K free namespaces that could be effected by the currently planned enforcement limit of 5 users.

FYI

  • This does MR not add the BackFill worker to the cron config. That will be done in a separate MR

    • Currently (2023-02-03) reconsidering this since the OverLimitNotificationWorker is guarded by a feature flag anyway.

  • CI build for this MR will be broken until Add mailer with templates (!95865 - merged) is merged and this branch is rebased on master. It's missing the mailer method on Notify to let the spec pass.

Screenshots

Updated mail looks like this now: image
(local) Email sent through Limited::Capacity worker processing namespaces Screenshot_2023-02-03_at_11.07.31
(local) group's details after being processed.
  • free_user_cap_over_limit_notified_at: Fri, 03 Feb 2023 11:00:33.063773000 UTC +00:00 was updated with when the group was notified
  • next_over_limit_check_at: Sat, 04 Feb 2023 11:00:32.000000000 UTC +00:00> scheduled to be checked again in ~24 hours

Screenshot_2023-02-03_at_13.50.34

(local) OverLimitNotificationWorkers running in Sidekiq with count of `5` jobs as configured by MAX_RUNNING_JOBS Screenshot_2023-02-03_at_11.06.11

Migrations

Rollback and Migration output https://gitlab.com/gitlab-org/gitlab/-/jobs/3743781742 
$ scripts/db_tasks db:migrate:down VERSION=20221219112632
Running: `bundle exec rake db:migrate:down VERSION=20221219112632`
main: == 20221219112632 AddNextOverLimitCheckAtToNamespaceDetails: reverting ========
main: -- transaction_open?()
main:    -> 0.0000s
main: -- remove_column(:namespace_details, :next_over_limit_check_at)
main:    -> 0.0013s
main: == 20221219112632 AddNextOverLimitCheckAtToNamespaceDetails: reverted (0.0235s) 
$ SKIP_POST_DEPLOYMENT_MIGRATIONS=true scripts/db_tasks db:migrate
Running: `bundle exec rake db:migrate`
main: == 20221219112632 AddNextOverLimitCheckAtToNamespaceDetails: migrating ========
main: -- transaction_open?()
main:    -> 0.0000s
main: -- add_column(:namespace_details, :next_over_limit_check_at, :datetime_with_timezone, {:null=>true})
main:    -> 0.0012s
main: == 20221219112632 AddNextOverLimitCheckAtToNamespaceDetails: migrated (0.0227s)

https://gitlab.com/gitlab-org/gitlab/-/jobs/3743781735

$ scripts/validate_migration_schema
$ git diff --name-only --diff-filter=A master -- db/migrate db/post_migrate
db/migrate/20221219112632_add_next_over_limit_check_at_to_namespace_details.rb
$ scripts/db_tasks db:migrate:down VERSION=20221219112632
Running: `bundle exec rake db:migrate:down VERSION=20221219112632`
main: == 20221219112632 AddNextOverLimitCheckAtToNamespaceDetails: reverting ========
main: -- transaction_open?()
main:    -> 0.0000s
main: -- remove_column(:namespace_details, :next_over_limit_check_at)
main:    -> 0.0010s
main: == 20221219112632 AddNextOverLimitCheckAtToNamespaceDetails: reverted (0.0193s) 
$ scripts/db_tasks db:schema:dump
Running: `bundle exec rake db:schema:dump`
$ git diff master -- db/structure.sql
$ scripts/db_tasks db:migrate
Running: `bundle exec rake db:migrate`
main: == 20221219112632 AddNextOverLimitCheckAtToNamespaceDetails: migrating ========
main: -- transaction_open?()
main:    -> 0.0000s
main: -- add_column(:namespace_details, :next_over_limit_check_at, :datetime_with_timezone, {:null=>true})
main:    -> 0.0011s
main: == 20221219112632 AddNextOverLimitCheckAtToNamespaceDetails: migrated (0.0206s)

How to set up and validate locally

  • assume in these steps that X = 2 (or choose your own value)
  • apply the patch from !104415 (merged) to your config/initializers/1_settings.rb
  • enable free_user_cap feature flag => Feature.enable :free_user_cap
  • enable free_user_cap_over_user_limit_mails feature flag => Feature.enable :free_user_cap_over_user_limit_mails
  • create a free, private, top level group with more than X members
  • go to http://localhost:3000/admin/application_settings/general
    • ensure Dashboard enforcement limit is set to lower than X
  • ensure in rails console ::Gitlab::CurrentSettings.dashboard_limit_enabled? is true
  • goto sidekiq admin panel http://localhost:3000/admin/sidekiq/cron
    • find the free_user_cap_backfill_notification_jobs_worker row
    • click Enqueue Now for free_user_cap_backfill_notification_jobs_worker and confirm the dialog with OK
  • goto http://localhost:3000/admin/sidekiq/busy
    • observe: you should see 5 busy OverLimitNotificationWorker jobs
  • goto the Letter opener at http://localhost:3000/rails/letter_opener/
    • go get a cup of joe / i.e. wait a bit
    • click Refresh
    • observe: (eventually) you should see an email for every free, private, top level group that you have in the DB with more than X members

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by SAM FIGUEROA

Merge request reports