Draft: [skip ci] PoC: refresh project_authorizations from a durable queue
What this is
A working proof of concept for https://gitlab.com/groups/gitlab-org/-/work_items/24326, not for merging. It exists so developers, and their agents, can pull down the whole design running end to end before picking up one of the epic's issues. The production work ships as small separate MRs.
Why
project_authorizations is kept up to date by many Sidekiq workers queued from many call sites, and refreshes still get lost (Redis or Sidekiq failures, deduplication, bulk writes that skip callbacks) or cost far more than the change needs. In this PoC, every write that can change access queues its own refresh in the same database transaction, and one worker recalculates only what that change can reach.
Where things are
- Producers:
db/migrate/20261031000001–07. Theauthz_refresh_eventstable, then trigger functions and one trigger migration per source table (members,project_group_links,group_group_links,projects,namespaces). - Queue:
app/models/authz/project_authorizations/refresh_event.rb. Lease, merge rules, continuation and the hand-off marker. - Worker:
app/workers/authz/project_authorizations/refresh_worker.rbandapp/services/authz/project_authorizations/refresh_service.rb. - Calculations:
reach.rb;recalculate_user_service.rb(a user's reach, per-slice first pass);recalculate_project_service.rb(a project's users);expand_service.rb(namespace and share events).
- Writes:
write_batch_service.rb. Per-user advisory locks, the writer barrier, delete and insert, and notifications after commit. - Verifier:
verify_service.rbandverification_sample.rb, called fromAuthz::ReverifyProjectAuthorizationsCronWorker. - Rollout switches:
RefreshEvent.computed?andreplaces_sidekiq?, read at the old producers' call sites;- the old refresh services delegate to the new path under
authz_refresh_write_path.
- Benchmark tooling:
scripts/authz_refresh_queue_poc/.
Flags
| Flag | Type | What it does |
|---|---|---|
authz_refresh_write_path |
gitlab_com_derisk | Old refresh services write through the locked write path. Required before anything computes. |
authz_refresh_compute_projects |
gitlab_com_derisk | Project, project member, transfer and share-lock events are recalculated instead of dropped |
authz_refresh_compute_memberships |
gitlab_com_derisk | Group membership and share events are recalculated. Requires the projects flag. |
authz_refresh_sidekiq_projects, authz_refresh_sidekiq_memberships |
ops, default on | Keep the old Sidekiq producers running; turn off only after the matching compute flag |
authz_refresh_verifier |
gitlab_com_derisk | Runs the verifier in the reverification cron |
authz_refresh_high_concurrency |
ops | 16 worker slots instead of 4 |
How to set up and validate locally
The migrations add triggers to members, projects and namespaces, so use separate databases rather than your GDK's main ones.
- Check out the branch in its own worktree:
git fetch origin authz-refresh-queue-poc git worktree add ../authz-refresh-queue-poc origin/authz-refresh-queue-poc cd ../authz-refresh-queue-poc - Point it at separate databases, and reuse the GDK checkout's other config and test binaries:
GDK_GITLAB=../gitlab # your GDK's gitlab checkout sed -E 's/database: gitlabhq_(development|test)(_ci|_sec)?$/database: gitlabhq_\1\2_authzq/' \ $GDK_GITLAB/config/database.yml > config/database.yml ln -s $GDK_GITLAB/config/gitlab.yml config/gitlab.yml ln -s $GDK_GITLAB/config/cable.yml config/cable.yml mkdir -p tmp && ln -sfn $GDK_GITLAB/tmp/tests tmp/tests bundle installconfig/cable.ymlis needed, or specs that publish GraphQL subscriptions fail. - Create and load the databases.
db/structure.sqlalready contains the PoC's table and triggers.RAILS_ENV=development bin/rails db:create db:schema:load RAILS_ENV=test bin/rails db:create db:schema:load - Run the PoC specs:
bin/rspec spec/models/authz/project_authorizations spec/services/authz/project_authorizations \ spec/workers/authzspec/services/authz/project_authorizations/refresh_service_spec.rbis the end-to-end spec. It makes changes through bulk SQL, shares, transfers and project moves, drains the queue, and compares the table withGitlab::ProjectAuthorizations. All flags are on in specs exceptauthz_refresh_write_pathandauthz_refresh_verifier; specs that need them enable them. - Watch the triggers and the worker in a Rails console (
bin/rails console, against the separate development database):To see the old producers turn off as well, disableFeature.enable(:authz_refresh_write_path) Feature.enable(:authz_refresh_compute_projects) Feature.enable(:authz_refresh_compute_memberships) group = Group.first user = User.last GroupMember.where(source: group, user: user).delete_all # bypasses callbacks; the trigger still fires Authz::ProjectAuthorizations::RefreshEvent.queued.pluck(:target_type, :target_id, :target_id2) Authz::ProjectAuthorizations::RefreshEvent.update_all(visible_at: nil) # skip the 5 s member debounce Authz::ProjectAuthorizations::RefreshService.new(Gitlab::Metrics::RuntimeLimiter.new(10.minutes)).executeauthz_refresh_sidekiq_projectsandauthz_refresh_sidekiq_memberships.
How to benchmark against production-sized data
This needs a Database Lab clone that you can write to, with pg_stat_statements, and DATABASE_LAB_URL set. Share only aggregates; the clone holds real data.
- Copy the table and triggers from your local development database to the clone (point
PGHOST,PGPORTandLOCAL_DBat the separate database from step 2):LOCAL_DB=gitlabhq_development_authzq scripts/authz_refresh_queue_poc/clone_install.sh - Run scenarios. Each makes a real change on the clone, runs today's path and the new path from the same state, checks that both write the same rows, and undoes the change:
Time is reported as
S=scripts/authz_refresh_queue_poc/benchmark.rb bin/rails runner $S membership label=<user_id>:<group_id> ... bin/rails runner $S project label=<project_id>:<rows_to_remove> ... bin/rails runner $S share label=<group_group_link_id> ... bin/rails runner $S queue drop=300 compute=300 merge=500pg_stat_statementsexecution time plus Ruby CPU, because clone round trips are about 140 ms. Notifications are stubbed out for both paths. - Measure trigger overhead on
members; it runs in one transaction that is rolled back:psql "$DATABASE_LAB_URL" -f scripts/authz_refresh_queue_poc/trigger_overhead.sql - Remove the PoC from the clone afterwards:
psql "$DATABASE_LAB_URL" -f scripts/authz_refresh_queue_poc/clone_uninstall.sql
Known gaps
- Unmeasured:
- transfer drain time for large subtrees;
- load on the primary (every read goes there);
- the largest single reach through the per-slice path.
- Specs not written yet:
- real transfer, link, member, EE activate and LDAP services with the old producers off;
- a two-connection race spec.
- Not built:
- queue depth and age metrics;
- autovacuum settings for the queue table;
- cleanup of the old workers and the Redis lock;
- moving
ProjectAuthorizations::Changesand the per-project finders intoAuthz::ProjectAuthorizations.
- Lint: the benchmark script doesn't pass RuboCop, and the commit skipped lefthook.
Benchmark results and the rollout plan are in the epic.