Add workers to clear notified at flags
What does this MR do and why?
implements https://gitlab.com/gitlab-org/gitlab/-/issues/379195+
Checks the past notified groups after 24 hours to see if they are still over_limit?
- If they are still over the limit: reschedule a new check for 24 hours later
- If they not over the limit anymore: clear the notification flag, so they can be notified again if they happen to go over the limit
Note for Infra-Engineers: Implemented with LimitedCapacity::Worker
Screenshots or screen recordings
Manipulating row in rails console, reloading the row and then waiting a bit reading the row again shows it was picked up by the worker and the over limit notified flag was cleared:
Database
The worker uses the following query to fetch new items to work on from the DB. This query will need an index to be created to run efficiently. I suggest we use a similar approach to the OverLimitNotification worker issue where we added an index as a follow-up blocking the feature flag roll out.
Without Index: Fetch a namespace_details item locked for update to work on **Time: 1.603 min**
UPDATE
"namespace_details"
SET
"next_over_limit_check_at" = to_timestamp(1678808868)
WHERE
"namespace_details"."namespace_id" IN (
SELECT
"namespace_details"."namespace_id"
FROM
"namespace_details"
WHERE
"namespace_details"."free_user_cap_over_limit_notified_at" <= '2023-03-12 15:47:48.266485'
ORDER BY
"namespace_details"."next_over_limit_check_at" ASC NULLS FIRST
LIMIT
1 FOR
UPDATE
SKIP LOCKED
) RETURNING *
Time: 1.603 min
- planning: 1.577 ms
- execution: 1.603 min
- I/O read: 1.099 min
- I/O write: 8.772 s
Shared buffers:
- hits: 34361 (~268.40 MiB) from the buffer pool
- reads: 661973 (~5.10 GiB) from the OS file cache, including disk I/O
- dirtied: 163431 (~1.20 GiB)
- writes: 42652 (~333.20 MiB)
WITH index: Fetch a namespace_details item locked for update to work on **Time: 3.153 ms**
Query same as above.
Time: 3.153 ms
- planning: 2.615 ms
- execution: 0.538 ms
- I/O read: 0.232 ms
- I/O write: 0.000 ms
Shared buffers:
- hits: 3 (~24.00 KiB) from the buffer pool
- reads: 4 (~32.00 KiB) from the OS file cache, including disk I/O
- dirtied: 0
- writes: 0
How to set up and validate locally
- Enable
free_user_cap
feature flag =>Feature.enable :free_user_cap
- Go to http://localhost:3000/admin/application_settings/general
- Set
Dashboard enforcement limit
to5
- Set
- Ensure in rails console
::Gitlab::CurrentSettings.dashboard_limit_enabled?
istrue
- Enable feature flag free_user_cap_clear_over_limit_notification_flags
Feature.enable :free_user_cap_clear_over_limit_notification_flags
- Create a top-level, free, private group, without a trial
- Edit the group's namespace details to have a timestamp in
free_user_cap_over_limit_notified_at
that is more than 24 hours ago and add a timestamp tonext_over_limit_check_at
that is more than 24 hours ago. (see screenshot) - Wait 0-10 min. Latest after the next 5th minute is passed the worker should have picked up your group and removed the timestamp in
free_user_cap_over_limit_notified_at
since it should not be over the user limit
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.