## Summary Roll out [deferring the project authorization safety-net workers on high main database WAL rate](https://gitlab.com/gitlab-org/gitlab/-/issues/626462), introduced in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/254347 and currently behind the `defer_safety_net_auth_refresh_on_wal_rate` feature flag. The flag covers these workers. Each one defers for 5 minutes, based on the main (`gitlab_main_org`) database WAL rate indicator only: - `Authz::ReverifyProjectAuthorizationsCronWorker` - `AuthorizedProjectUpdate::PeriodicRecalculateWorker` - `AuthorizedProjectUpdate::UserRefreshOverUserRangeWorker` The primary refresh workers are behind a separate flag, `defer_primary_auth_refresh_on_wal_rate`, tracked in https://gitlab.com/gitlab-org/gitlab/-/work_items/632253. - DRI: @hmehra - Team Slack channel: `#g_sscs_authorization` > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? - **Main risk:** during a long WAL rate spike, the safety net pauses. The reverification queue grows, and the periodic safety-net sweep is delayed. Users then wait longer for stale access to be corrected, whether that is access they should have lost or access they should have gained. The primary authorization refresh path is not affected by this flag. - **What to watch:** for the reverification cron only, the job-done log fields `queue_lag_s` and `queue_drained`. Other deferred jobs show `job_status: deferred` and `job_deferred_by: database_health_check` in Sidekiq logs. - **Rollback:** instant. Disable the flag and the workers stop deferring. ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#g_sscs_authorization`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). The flag is checked with the `:instance` actor, so it is either on or off for the whole instance. Percentage and per-actor rollout do not apply. Enable this flag and confirm it is stable before enabling `defer_primary_auth_refresh_on_wal_rate`. **Non-production** ``` /chatops gitlab run feature set defer_safety_net_auth_refresh_on_wal_rate true --dev --pre --staging --staging-ref ``` **Production** ``` /chatops gitlab run feature set defer_safety_net_auth_refresh_on_wal_rate true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete defer_safety_net_auth_refresh_on_wal_rate --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set defer_safety_net_auth_refresh_on_wal_rate false # production /chatops gitlab run feature set defer_safety_net_auth_refresh_on_wal_rate false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete defer_safety_net_auth_refresh_on_wal_rate --dev --pre --staging --staging-ref --production # remove entirely ```