Skip no-op todo cleanup jobs for projects the user can still access

What does this MR do and why?

When a member loses access to a group, for example when they are lowered to Guest or removed, TodosDestroyer::EntityLeaveWorker runs Todos::Destroy::EntityLeaveService. For a non-private group, the service enqueues one TodosDestroyer::PrivateFeaturesWorker job for every project in the group hierarchy.

Each of those jobs runs Todos::Destroy::UnauthorizedFeaturesService, which returns early when the user still has any authorization on the project. A member lowered to Guest keeps access to every project in the hierarchy, so for a large group almost all of these jobs do nothing, but they still fill the queue.

This MR applies the same check before enqueueing: the service only enqueues PrivateFeaturesWorker for projects where the user has no project authorization left, and enqueues them in batches with bulk_perform_async. A downgrade to Guest no longer enqueues any PrivateFeaturesWorker job. A removal still enqueues jobs for the projects the user can no longer access, which are the only projects where the worker deletes todos, so once authorizations have settled, the todos that get deleted do not change.

The change is behind the todos_private_features_only_unauthorized_projects feature flag, disabled by default.

Database

EntityLeaveService#enqueue_private_features_worker filters the existing projects scope with a correlated NOT EXISTS on the user's project authorizations (Guest or higher), and iterates the filtered scope with each_batch. It runs in the low-urgency TodosDestroyer::EntityLeaveWorker job, not in a web request.

Each run makes three kinds of queries: the first batch boundary, the next batch boundary, and the batch query. After a downgrade to Guest, the first boundary query finds no rows, so it is the only query that runs.

The plans use gitlab-org (about 9,400 projects in the hierarchy) and three users: a Guest of gitlab-org with 30k authorizations, a user with 124k authorizations (1.2k in gitlab-org), and a user with no authorizations, as after a removal. Every query is under 100 ms warm:

User Query Rows Warm Plan
Guest, 30k authorizations First boundary 0 48.8 ms https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/90184/commands/196428
124k authorizations First boundary 1 86.9 ms https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/90184/commands/196430
124k authorizations Next boundary 1 72.0 ms https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/90184/commands/196432
124k authorizations Batch 1,000 49.1 ms https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/90184/commands/196434
No authorizations First boundary 1 47.0 ms https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/90184/commands/196436
No authorizations Next boundary 1 36.5 ms https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/90184/commands/196438
No authorizations Batch 1,000 40.7 ms https://console.postgres.ai/gitlab/projects/gitlab-production-main/sessions/90184/commands/196440

Cold, the first runs were slower: 855 ms for the batch query of the user with no authorizations (805 ms of it reading 1,003 project_authorizations index pages, one per project in the batch), and 1.06 s and 3.83 s for the first boundary queries of the 124k and Guest users.

The plan links are the first runs, and the warm column is a second run of the same query. Both runs of each query are also listed in an internal note: !268756 (comment 4224479656)

SQL
-- 1. First batch boundary
SELECT "projects"."id" FROM "projects" WHERE "projects"."namespace_id" IN (…existing projects scope for <group_id> and <user_id>…) AND (NOT EXISTS (SELECT 1 FROM "project_authorizations" WHERE "project_authorizations"."user_id" = <user_id> AND (project_authorizations.project_id = projects.id) AND (project_authorizations.access_level >= 10))) ORDER BY "projects"."id" ASC LIMIT 1

-- 2. Next batch boundary
… AND (NOT EXISTS (…)) AND "projects"."id" >= <batch_start> ORDER BY "projects"."id" ASC LIMIT 1 OFFSET 1000

-- 3. Batch query
… AND (NOT EXISTS (…)) AND "projects"."id" >= <batch_start> AND "projects"."id" < <batch_end>

The NOT IN (WITH "groups_with_min_access_level_cte" …) inside the existing projects scope is not the new filter. It drops groups where the user is Planner or higher, as before this change.

The boundary queries scan the rest of the hierarchy before they pick an id, so their cost follows the hierarchy size. The batch query probes project_authorizations once per project in the batch. Before this change, projects.each ran the projects scope once. With each_batch it runs once per boundary and batch query, in exchange for bounded batches. For gitlab-org after a removal that is 21 queries: the first boundary, then a next boundary and a batch query for each of the 10 batches.

References

Screenshots or screen recordings

Not applicable, no UI changes.

How to set up and validate locally

  1. In the Rails console, create a group with a few projects and a user who is a Developer of the group:

    group = FactoryBot.create(:group, :internal)
    projects = FactoryBot.create_list(:project, 3, :internal, group: group)
    user = FactoryBot.create(:user)
    group.add_developer(user)
  2. Lower the user to Guest, then run the service and count the enqueued jobs:

    group.members.find_by(user: user).update!(access_level: Gitlab::Access::GUEST)
    
    require 'sidekiq/testing'
    Sidekiq::Testing.fake! do
      Todos::Destroy::EntityLeaveService.new(user.id, group.id, 'Group').execute
      TodosDestroyer::PrivateFeaturesWorker.jobs.size
    end
  3. With the flag disabled, one job is enqueued for each project. Enable the flag and run it again: no jobs are enqueued.

    Feature.enable(:todos_private_features_only_unauthorized_projects)
  4. Remove the user from the group and run the service again: one job is enqueued for each project the user can no longer access.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Agnes Slota

Merge request reports

Loading
Loading