Add workers to notify over limit owners
What does this MR do and why?
-
Part of #367549+
-
Add a worker to be triggered by cron(settings for the cron will come in a follow-up MR), to schedule another worker that sends an email notification to the group owner that's gone over the free user limit. Both observe if the
dashboard_limit
is activated, and are also guarded by the activation of thefree_user_cap_over_user_limit_mails
feature flag. -
Add a finder to query the DB for groups that could potentially be over limit according to the free user criteria. Should make it easy to adapt the parameters if we pivot to another method for determining the cap.
Performance considerations
- We've changed the job
OverLimitNotificationWorker
to implement the Limited::Capacity concern with a current max jobs of5
and a batch size of1
. It will keep processing groups until it has reached processed all unprocessed groups that apply to the enforcement criteria (free, top-level, and private).- The constants
MAX_RUNNING_JOBS
&BATCH_LIMIT
are used to scale the concurrency (up or down)
- The constants
- The jobs are guarded by a feature flag
free_user_cap_over_user_limit_mails
- The mailer job will silently exit unless the flag is set
over_limit?
check
meta: Work on improving performance of the over_limit was completed in https://gitlab.com/gitlab-org/gitlab/-/issues/383264+ and has significantly reduced the query times for this check.
-
It has been running smoothly on normal page request cycles
-
Planning & Execution Analysis for
billed_group_users
(part ofover_limit?
check)- using GitLab's group ID
9970
as an example for a very large top level group-
https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/15260/commands/53126
- Time: ~40 ms
- planning: ~2 ms
- execution: ~38 ms
- Time: ~40 ms
-
https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/15260/commands/53126
- using group ID
1937166
as an example with a group with71
traversals-
https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/15260/commands/53132
- Time: ~14 ms
- planning: ~2 ms
- execution: ~12 ms
- Time: ~14 ms
-
https://postgres.ai/console/gitlab/gitlab-production-tunnel-pg12/sessions/15260/commands/53132
- using GitLab's group ID
Potential Namespaces Effected
- This Sisense Board reads roughly
100K
free namespaces that could be effected by the currently planned enforcement limit of5
users.
FYI
-
This does MR not add the BackFill worker to the cron config. That will be done in a separate MR
- Currently (2023-02-03) reconsidering this since the
OverLimitNotificationWorker
is guarded by a feature flag anyway.
- Currently (2023-02-03) reconsidering this since the
-
CI build for this MR will be broken until Add mailer with templates (!95865 - merged) is merged and this branch is rebased on master. It's missing the mailer method onNotify
to let the spec pass.
Screenshots
(local) group's details after being processed.
-
free_user_cap_over_limit_notified_at: Fri, 03 Feb 2023 11:00:33.063773000 UTC +00:00
was updated with when the group was notified -
next_over_limit_check_at: Sat, 04 Feb 2023 11:00:32.000000000 UTC +00:00>
scheduled to be checked again in ~24 hours
(local) OverLimitNotificationWorkers running in Sidekiq with count of `5` jobs as configured by MAX_RUNNING_JOBS
![Screenshot_2023-02-03_at_11.06.11](/-/project/278964/uploads/991bf95cc78bbbe54618366c7330ed80/Screenshot_2023-02-03_at_11.06.11.png)
Migrations
Rollback and Migration output
https://gitlab.com/gitlab-org/gitlab/-/jobs/3743781742$ scripts/db_tasks db:migrate:down VERSION=20221219112632
Running: `bundle exec rake db:migrate:down VERSION=20221219112632`
main: == 20221219112632 AddNextOverLimitCheckAtToNamespaceDetails: reverting ========
main: -- transaction_open?()
main: -> 0.0000s
main: -- remove_column(:namespace_details, :next_over_limit_check_at)
main: -> 0.0013s
main: == 20221219112632 AddNextOverLimitCheckAtToNamespaceDetails: reverted (0.0235s)
$ SKIP_POST_DEPLOYMENT_MIGRATIONS=true scripts/db_tasks db:migrate
Running: `bundle exec rake db:migrate`
main: == 20221219112632 AddNextOverLimitCheckAtToNamespaceDetails: migrating ========
main: -- transaction_open?()
main: -> 0.0000s
main: -- add_column(:namespace_details, :next_over_limit_check_at, :datetime_with_timezone, {:null=>true})
main: -> 0.0012s
main: == 20221219112632 AddNextOverLimitCheckAtToNamespaceDetails: migrated (0.0227s)
https://gitlab.com/gitlab-org/gitlab/-/jobs/3743781735
$ scripts/validate_migration_schema
$ git diff --name-only --diff-filter=A master -- db/migrate db/post_migrate
db/migrate/20221219112632_add_next_over_limit_check_at_to_namespace_details.rb
$ scripts/db_tasks db:migrate:down VERSION=20221219112632
Running: `bundle exec rake db:migrate:down VERSION=20221219112632`
main: == 20221219112632 AddNextOverLimitCheckAtToNamespaceDetails: reverting ========
main: -- transaction_open?()
main: -> 0.0000s
main: -- remove_column(:namespace_details, :next_over_limit_check_at)
main: -> 0.0010s
main: == 20221219112632 AddNextOverLimitCheckAtToNamespaceDetails: reverted (0.0193s)
$ scripts/db_tasks db:schema:dump
Running: `bundle exec rake db:schema:dump`
$ git diff master -- db/structure.sql
$ scripts/db_tasks db:migrate
Running: `bundle exec rake db:migrate`
main: == 20221219112632 AddNextOverLimitCheckAtToNamespaceDetails: migrating ========
main: -- transaction_open?()
main: -> 0.0000s
main: -- add_column(:namespace_details, :next_over_limit_check_at, :datetime_with_timezone, {:null=>true})
main: -> 0.0011s
main: == 20221219112632 AddNextOverLimitCheckAtToNamespaceDetails: migrated (0.0206s)
How to set up and validate locally
- assume in these steps that
X = 2
(or choose your own value) - apply the patch from !104415 (merged) to your
config/initializers/1_settings.rb
- enable
free_user_cap
feature flag =>Feature.enable :free_user_cap
- enable
free_user_cap_over_user_limit_mails
feature flag =>Feature.enable :free_user_cap_over_user_limit_mails
- create a free, private, top level group with more than
X
members - go to http://localhost:3000/admin/application_settings/general
- ensure
Dashboard enforcement limit
is set to lower thanX
- ensure
- ensure in rails console
::Gitlab::CurrentSettings.dashboard_limit_enabled?
istrue
- goto sidekiq admin panel http://localhost:3000/admin/sidekiq/cron
- find the
free_user_cap_backfill_notification_jobs_worker
row - click
Enqueue Now
forfree_user_cap_backfill_notification_jobs_worker
and confirm the dialog withOK
- find the
- goto http://localhost:3000/admin/sidekiq/busy
- observe: you should see
5
busyOverLimitNotificationWorker
jobs
- observe: you should see
- goto the Letter opener at http://localhost:3000/rails/letter_opener/
- go get a cup of joe / i.e. wait a bit
- click
Refresh
- observe: (eventually) you should see an email for every free, private, top level group that you have in the DB with more than
X
members
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.