Skip to content

Reduce to single over limit notification worker

Sam Figueroa requested to merge sf/fix/overlimit-notifiaciton-worker-concurrency-reduction into master

What does this MR do and why?

Reduce the concurrency of running OverlimitNotificationWorker jobs to 1 to simplify observation of CPU load.

Based on https://gitlab.com/gitlab-org/gitlab/-/issues/413547 we have also increased the batch size to 100.

Urgency context: Due to free user limits rolling out this week to existing namespaces, there is a desire to have this change(or whatever comes of it after reviews) get to production in the next few days.

Database Details

Due to the batch size increasing, I have tested the query in postgres.ai to get an idea of how it performs. This can be seen here. While the time is Time: 682.004 ms, it's worth noting previous runs where ~1s.

See formatted SQL below. This was obtained by running:

y Namespaces::FreeUserCap::OverLimitNotificationWorker.new.next_batch_sql

in the rails console after updating the batch size (and making the method public). Then it was formatted with https://sqlformat.darold.net/.

Formatted SQL 
UPDATE
    "namespace_details"
SET
    "next_over_limit_check_at" = to_timestamp(1685701326)
WHERE
    "namespace_details"."namespace_id" IN (
        SELECT
            "namespace_details"."namespace_id"
        FROM
            "namespace_details"
        WHERE
            "namespace_details"."free_user_cap_over_limit_notified_at" IS NULL
            AND ("namespace_details"."next_over_limit_check_at" <= '2023-06-01 10:22:06.688936'
                OR "namespace_details"."next_over_limit_check_at" IS NULL)
            AND "namespace_details"."namespace_id" IN (
                SELECT
                    "namespaces"."id"
                FROM
                    "namespaces"
                LEFT OUTER JOIN "gitlab_subscriptions" ON "gitlab_subscriptions"."namespace_id" = "namespaces"."id"
            LEFT OUTER JOIN "plans" ON "plans"."id" = "gitlab_subscriptions"."hosted_plan_id"
        WHERE
            "namespaces"."type" = 'Group'
            AND ("plans"."name" IN ('default', 'free')
                OR "plans"."name" IS NULL)
            AND "namespaces"."parent_id" IS NULL
            AND "namespaces"."visibility_level" != 20)
    ORDER BY
        "namespace_details"."next_over_limit_check_at" ASC NULLS FIRST
    LIMIT 100
    FOR UPDATE
        SKIP LOCKED)
RETURNING
    *;

Screenshots or screen recordings

How to set up and validate locally

  1. Enable free_user_cap_clear_over_limit_notification_flags feature flag
  2. check localhost:3000/admin/sidekiq for job stats

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Doug Stull

Merge request reports