<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Label this issue](https://contributors.gitlab.com/manage-issue?action=label&projectId=278964&issueIid=617402) </details> <!--IssueSummary end--> <!--Title suggestion: [FF] `use_db_to_queue_safety_net_auth_refresh` -- <short description>--> ## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/249045) currently behind the `use_db_to_queue_safety_net_auth_refresh `feature flag. - DRI: @hmehra - Team Slack channel: `g_scss_authorization` ## What could go wrong? ### Monitor 1. [Grafana dashboard](https://dashboards.gitlab.net/goto/dfyvnqcrs3474e?orgId=1) 2. Queue drained - Filter for `json.class: "Authz::ReverifyProjectAuthorizationsCronWorker" and json.job_status: done` - Look for `json.extra.authz_reverify_project_authorizations_cron_worker.queue_drained` and `json.extra.authz_reverify_project_authorizations_cron_worker.users_processed` - [Production logs](https://log.gprd.gitlab.net/app/r/s/QtukX) - If `queue_drained = false` for a long period of time, increase capacity - https://gitlab.com/gitlab-org/gitlab/-/merge_requests/258864 - [Queue lag visualisation (in seconds)](https://log.gprd.gitlab.net/app/r/s/4uqTO) 3. `Duration` -\> target duration should not exceed 300s 4. Monitor safety net additions / deletions - they are only logged when the refresh actually changes rows - [Production logs](https://log.gprd.gitlab.net/app/r/s/qsdmV) 5. Queue size - number of rows in the table - increase concurrency limit if it keeps growing and the rows are not getting processed - `SELECT COUNT(*) FROM project_authorization_reverifications WHERE status IN (0,2)` 6. Queue age - number of rows in the table have been pending for more than 24 hours - `SELECT COUNT(*) WHERE FROM project_authorization_reverifications status IN (0, 2) AND enqueued_at < now() - interval '24 hours'` 7. Job health - any sustained failure rate i.e. `json.job_status: fail` 8. Cronjob shard - the new 5 concurrent CPU-heavy 270s jobs can starve other cron workers - [Sidekiq shards dashboard](https://dashboards.gitlab.net/goto/ffyvnjxyncnb4d?orgId=1) 9. When the FF is 100% enabled, calls to `UserRefreshFromReplicaWorker` should go to 0 ### Rollback 1. If the feature flag is enabled then disabled, then users that were queued to be refreshed while the FF was enabled will continue to be refreshed. 2. If we want all safety nets to be stopped we can enable `do_not_run_safety_net_auth_refresh_jobs` <!--Optional but recommended: blast radius, data-loss risk, and the dashboard(s) you'll watch on https://dashboards.gitlab.net. Delete if not applicable.--> ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#g_scss_authorization`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set <feature-flag-name> 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set <feature-flag-name> true --dev --pre --staging --staging-ref ``` **Production** — percentage rollout (wait ≥15 min between steps, watch dashboards): ``` /chatops gitlab run feature set <feature-flag-name> <percentage> --actors ``` Or target specific actors instead: ``` /chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss <feature-flag-name> true /chatops gitlab run feature set --group=gitlab-org,gitlab-com <feature-flag-name> true /chatops gitlab run feature set --user=<gitlab-username-of-dri> <feature-flag-name> true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete <feature-flag-name> --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set <feature-flag-name> false # production /chatops gitlab run feature set <feature-flag-name> false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete <feature-flag-name> --dev --pre --staging --staging-ref --production # remove entirely ``` <!--Uncomment the appropriate type label /label ~"type::feature" ~"feature::addition" /label ~"type::maintenance" /label ~"type::bug"-->