<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Label this issue](https://contributors.gitlab.com/manage-issue?action=label&projectId=278964&issueIid=607948) </details> <!--IssueSummary end--> <!-- Title suggestion: [FF] `deprioritize_destroyed_project_bot_user_project_authorizations_refresh` -- refresh project authorizations with low urgency when a project bot user is destroyed --> ## Summary Roll out [the feature](https://gitlab.com/groups/gitlab-org/-/epics/22633) currently behind the `deprioritize_destroyed_project_bot_user_project_authorizations_refresh` feature flag. - DRI: @eugielimpin - Team Slack channel: `#g_sscs_authorization` - Introduced by: gitlab-org/gitlab!248268 (19.3) When the flag is enabled, destroying a **project bot** user schedules its per-membership project authorizations refresh with `UserProjectAccessChangedService::MEDIUM_PRIORITY`, which routes the work to `AuthorizedProjectUpdate::UserRefreshWithLowUrgencyWorker` (low urgency, batched through `bulk_perform_in`) instead of the high-urgency `AuthorizedProjectsWorker`. Nothing else changes: non-bot user deletion and every other membership change keep the existing high-urgency path. The feature actor is the **user being destroyed**, so a percentage rollout is a percentage of destroyed project bot users, not of the people or jobs triggering the deletion. ### Strategy - [ ] Rollout to 25% of actors - [ ] Rollout to 50% of actors - [ ] Rollout to 75% of actors - [ ] Rollout globally The bulk of the affected traffic comes from the nightly `ResourceAccessTokens::InactiveTokensDeletionCronWorker` run, so allow at least one nightly cycle per step rather than only the 15-minute minimum. > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What we expect to happen? Fewer high-urgency `AuthorizedProjectsWorker` runs exceeding the 10s target duration during the nightly inactive-token bot cleanup, and lower scheduling latency on the urgent authorized-projects shard: ``` # Data view: pubsub-sidekiq-inf-gprd json.class:"AuthorizedProjectsWorker" AND json.meta.root_caller_id:"ResourceAccessTokens::InactiveTokensDeletionCronWorker" AND json.duration_s>10 ``` A matching increase should show up on the low-urgency worker: ``` # Data view: pubsub-sidekiq-inf- json.class:"AuthorizedProjectUpdate::UserRefreshWithLowUrgencyWorker" AND json.meta.root_caller_id:"ResourceAccessTokens::InactiveTokensDeletionCronWorker" ``` ## What could go wrong? - **Delayed row removal.** A destroyed project bot's `project_authorizations` rows stay in place longer than before, until the low-urgency refresh runs (or until the user record itself is deleted, which removes them anyway). This is not an access risk: a bot only reaches projects through a resource access token, `GlobalPolicy` denies `:log_in` for it regardless of token state, and `Users::DestroyService` blocks the user before destroying its memberships. - **Low-urgency shard load.** The work moves rather than disappears. Watch the `authorized_project_update` queue namespace for queue growth if the nightly cleanup is large. - Dashboards to watch on <https://dashboards.gitlab.net>: Sidekiq worker detail for `AuthorizedProjectsWorker` and `AuthorizedProjectUpdate::UserRefreshWithLowUrgencyWorker`, and the Sidekiq shard detail for the urgent authorized-projects shard (apdex, queue length, scheduling latency). ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#g_sscs_authorization`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set deprioritize_destroyed_project_bot_user_project_authorizations_refresh 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set deprioritize_destroyed_project_bot_user_project_authorizations_refresh true --dev --pre --staging --staging-ref ``` **Production** — percentage rollout (wait ≥15 min between steps, watch dashboards): ``` /chatops gitlab run feature set deprioritize_destroyed_project_bot_user_project_authorizations_refresh <percentage> --actors ``` Actor targeting is not practical here: the actor is the project bot user being destroyed (`project_<id>_bot_<hash>`), not a person, group, or project. Use the percentage rollout above. ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated — n/a, no user-facing documented behaviour changes - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any — n/a - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable — n/a ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete deprioritize_destroyed_project_bot_user_project_authorizations_refresh --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set deprioritize_destroyed_project_bot_user_project_authorizations_refresh false # production /chatops gitlab run feature set deprioritize_destroyed_project_bot_user_project_authorizations_refresh false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete deprioritize_destroyed_project_bot_user_project_authorizations_refresh --dev --pre --staging --staging-ref --production # remove entirely ```