Draft: [skip ci] PoC: refresh project_authorizations from a durable queue

What this is

A working proof of concept for https://gitlab.com/groups/gitlab-org/-/work_items/24326, not for merging. It exists so developers, and their agents, can pull down the whole design running end to end before picking up one of the epic's issues. The production work ships as small separate MRs.

Why

project_authorizations is kept up to date by many Sidekiq workers queued from many call sites, and refreshes still get lost (Redis or Sidekiq failures, deduplication, bulk writes that skip callbacks) or cost far more than the change needs. In this PoC, every write that can change access queues its own refresh in the same database transaction, and one worker recalculates only what that change can reach.

Where things are

  • Producers: db/migrate/20261031000001–07. The authz_refresh_events table, then trigger functions and one trigger migration per source table (members, project_group_links, group_group_links, projects, namespaces).
  • Queue: app/models/authz/project_authorizations/refresh_event.rb. Lease, merge rules, continuation and the hand-off marker.
  • Worker: app/workers/authz/project_authorizations/refresh_worker.rb and app/services/authz/project_authorizations/refresh_service.rb.
  • Calculations:
    • reach.rb;
    • recalculate_user_service.rb (a user's reach, per-slice first pass);
    • recalculate_project_service.rb (a project's users);
    • expand_service.rb (namespace and share events).
  • Writes: write_batch_service.rb. Per-user advisory locks, the writer barrier, delete and insert, and notifications after commit.
  • Verifier: verify_service.rb and verification_sample.rb, called from Authz::ReverifyProjectAuthorizationsCronWorker.
  • Rollout switches:
    • RefreshEvent.computed? and replaces_sidekiq?, read at the old producers' call sites;
    • the old refresh services delegate to the new path under authz_refresh_write_path.
  • Benchmark tooling: scripts/authz_refresh_queue_poc/.

Flags

Flag Type What it does
authz_refresh_write_path gitlab_com_derisk Old refresh services write through the locked write path. Required before anything computes.
authz_refresh_compute_projects gitlab_com_derisk Project, project member, transfer and share-lock events are recalculated instead of dropped
authz_refresh_compute_memberships gitlab_com_derisk Group membership and share events are recalculated. Requires the projects flag.
authz_refresh_sidekiq_projects, authz_refresh_sidekiq_memberships ops, default on Keep the old Sidekiq producers running; turn off only after the matching compute flag
authz_refresh_verifier gitlab_com_derisk Runs the verifier in the reverification cron
authz_refresh_high_concurrency ops 16 worker slots instead of 4

How to set up and validate locally

The migrations add triggers to members, projects and namespaces, so use separate databases rather than your GDK's main ones.

  1. Check out the branch in its own worktree:
    git fetch origin authz-refresh-queue-poc
    git worktree add ../authz-refresh-queue-poc origin/authz-refresh-queue-poc
    cd ../authz-refresh-queue-poc
  2. Point it at separate databases, and reuse the GDK checkout's other config and test binaries:
    GDK_GITLAB=../gitlab   # your GDK's gitlab checkout
    sed -E 's/database: gitlabhq_(development|test)(_ci|_sec)?$/database: gitlabhq_\1\2_authzq/' \
      $GDK_GITLAB/config/database.yml > config/database.yml
    ln -s $GDK_GITLAB/config/gitlab.yml config/gitlab.yml
    ln -s $GDK_GITLAB/config/cable.yml config/cable.yml
    mkdir -p tmp && ln -sfn $GDK_GITLAB/tmp/tests tmp/tests
    bundle install
    config/cable.yml is needed, or specs that publish GraphQL subscriptions fail.
  3. Create and load the databases. db/structure.sql already contains the PoC's table and triggers.
    RAILS_ENV=development bin/rails db:create db:schema:load
    RAILS_ENV=test bin/rails db:create db:schema:load
  4. Run the PoC specs:
    bin/rspec spec/models/authz/project_authorizations spec/services/authz/project_authorizations \
      spec/workers/authz
    spec/services/authz/project_authorizations/refresh_service_spec.rb is the end-to-end spec. It makes changes through bulk SQL, shares, transfers and project moves, drains the queue, and compares the table with Gitlab::ProjectAuthorizations. All flags are on in specs except authz_refresh_write_path and authz_refresh_verifier; specs that need them enable them.
  5. Watch the triggers and the worker in a Rails console (bin/rails console, against the separate development database):
    Feature.enable(:authz_refresh_write_path)
    Feature.enable(:authz_refresh_compute_projects)
    Feature.enable(:authz_refresh_compute_memberships)
    
    group = Group.first
    user = User.last
    GroupMember.where(source: group, user: user).delete_all   # bypasses callbacks; the trigger still fires
    Authz::ProjectAuthorizations::RefreshEvent.queued.pluck(:target_type, :target_id, :target_id2)
    
    Authz::ProjectAuthorizations::RefreshEvent.update_all(visible_at: nil)   # skip the 5 s member debounce
    Authz::ProjectAuthorizations::RefreshService.new(Gitlab::Metrics::RuntimeLimiter.new(10.minutes)).execute
    To see the old producers turn off as well, disable authz_refresh_sidekiq_projects and authz_refresh_sidekiq_memberships.

How to benchmark against production-sized data

This needs a Database Lab clone that you can write to, with pg_stat_statements, and DATABASE_LAB_URL set. Share only aggregates; the clone holds real data.

  1. Copy the table and triggers from your local development database to the clone (point PGHOST, PGPORT and LOCAL_DB at the separate database from step 2):
    LOCAL_DB=gitlabhq_development_authzq scripts/authz_refresh_queue_poc/clone_install.sh
  2. Run scenarios. Each makes a real change on the clone, runs today's path and the new path from the same state, checks that both write the same rows, and undoes the change:
    S=scripts/authz_refresh_queue_poc/benchmark.rb
    bin/rails runner $S membership label=<user_id>:<group_id> ...
    bin/rails runner $S project label=<project_id>:<rows_to_remove> ...
    bin/rails runner $S share label=<group_group_link_id> ...
    bin/rails runner $S queue drop=300 compute=300 merge=500
    Time is reported as pg_stat_statements execution time plus Ruby CPU, because clone round trips are about 140 ms. Notifications are stubbed out for both paths.
  3. Measure trigger overhead on members; it runs in one transaction that is rolled back:
    psql "$DATABASE_LAB_URL" -f scripts/authz_refresh_queue_poc/trigger_overhead.sql
  4. Remove the PoC from the clone afterwards:
    psql "$DATABASE_LAB_URL" -f scripts/authz_refresh_queue_poc/clone_uninstall.sql

Known gaps

  • Unmeasured:
    • transfer drain time for large subtrees;
    • load on the primary (every read goes there);
    • the largest single reach through the per-slice path.
  • Specs not written yet:
    • real transfer, link, member, EE activate and LDAP services with the old producers off;
    • a two-connection race spec.
  • Not built:
    • queue depth and age metrics;
    • autovacuum settings for the queue table;
    • cleanup of the old workers and the Redis lock;
    • moving ProjectAuthorizations::Changes and the per-project finders into Authz::ProjectAuthorizations.
  • Lint: the benchmark script doesn't pass RuboCop, and the commit skipped lefthook.

Benchmark results and the rollout plan are in the epic.

Merge request reports

Loading
Loading