The 20 feature flags introduced for the cohort 1-6 Labkit rate limiter migrations should be removed from the codebase once all cohorts are fully enabled and stable in production. > ⚠️ **Cohort 5 is still mid-rollout** (gitlab-com/gl-infra/production-engineering#28812, `workflow::in dev`): its `*_db_duration_*` keys are still in shadow (not enforced) as of 2026-06-09. Its two flags must **not** be removed until its enforcement soak completes. Cohorts 1-4 and 6 are enabled and enforcing on .com. ## Flags to remove (20 total) ### Cohort 1 — per-key flags (10) — rollout issue gitlab-com/gl-infra/production-engineering#28803 #### Labkit routing flags - `rate_limiter_use_labkit_pipelines_create` - `rate_limiter_use_labkit_notes_create` - `rate_limiter_use_labkit_search_rate_limit` - `rate_limiter_use_labkit_users_get_by_id` - `rate_limiter_use_labkit_user_sign_in` #### Enforcement flags - `rate_limiter_use_labkit_pipelines_create_enforce` - `rate_limiter_use_labkit_notes_create_enforce` - `rate_limiter_use_labkit_search_rate_limit_enforce` - `rate_limiter_use_labkit_users_get_by_id_enforce` - `rate_limiter_use_labkit_user_sign_in_enforce` ### Cohort 2 — cohort-wide flags (2) — rollout issue gitlab-com/gl-infra/production-engineering#28809 - `rate_limiter_use_labkit_cohort_2` - `rate_limiter_use_labkit_cohort_2_enforce` ### Cohort 3 — cohort-wide flags (2) — rollout issue gitlab-com/gl-infra/production-engineering#28810 - `rate_limiter_use_labkit_cohort_3` - `rate_limiter_use_labkit_cohort_3_enforce` ### Cohort 4 — cohort-wide flags (2) — rollout issue gitlab-com/gl-infra/production-engineering#28811 - `rate_limiter_use_labkit_cohort_4` - `rate_limiter_use_labkit_cohort_4_enforce` ### Cohort 5 — cohort-wide flags (2) — rollout issue gitlab-com/gl-infra/production-engineering#28812 - `rate_limiter_use_labkit_cohort_5` - `rate_limiter_use_labkit_cohort_5_enforce` > Blocked: remove these two only after cohort 5's enforcement flip and soak are complete. Do not include them in the cleanup MR or ChatOps run until then. ### Cohort 6 — cohort-wide flags (2) — rollout issue gitlab-com/gl-infra/production-engineering#29077 - `rate_limiter_use_labkit_cohort_6` - `rate_limiter_use_labkit_cohort_6_enforce` ## .com enablement state - Cohort 1 + 2: fully enabled 2026-05-08 (per @mwoolf on gitlab-org/gitlab#598560) — soak window passed - Cohort 3: fully enabled 2026-05-15 (per @mwoolf on gitlab-org/gitlab#599632) — @mwoolf signed off on Slack to ship without the full 7-day soak - Cohort 4: enabled and enforcing on .com (no shadow divergence signal in Grafana as of 2026-06-09); confirm the exact enablement date in gitlab-com/gl-infra/production-engineering#28811 before deletion - Cohort 5: **not yet enforced** — still in shadow on .com (gitlab-com/gl-infra/production-engineering#28812, `workflow::in dev`). Hold flag removal. - Cohort 6: enabled and enforcing on .com (no shadow divergence signal in Grafana as of 2026-06-09); confirm the exact enablement date in gitlab-com/gl-infra/production-engineering#29077 before deletion ## Steps 1. Confirm the flags eligible for removal (18 now; the 2 cohort 5 flags only once cohort 5 is enforced) are fully enabled (`true`, no actor percentage) and stable in production 2. Confirm the cohort migration MRs have reached `workflow::production`, including the late additions gitlab-org/gitlab!239752 (`token_exchange`, cohort 6) and gitlab-org/gitlab!239753 (`project_generate_new_export`, cohort 2) 3. Spot-check Grafana: `gitlab_rate_limiter_labkit_shadow_total{agreement="diverge", boundary="false"}` should be flat across all cohort keys 4. Remove `Feature.enabled?` guards from application code in `gitlab-org/gitlab` and collapse `LabkitAdapter` to always-on 5. Delete the eligible feature flag YAML files from `config/feature_flags/wip/` 6. Open a single unified MR in `gitlab-org/gitlab` 7. After the MR reaches `workflow::production`, run the ChatOps cleanup below ## ChatOps cleanup (after MR is on `workflow::production`) Run in `#production`: ``` /chatops gitlab run feature delete rate_limiter_use_labkit_pipelines_create --dev --pre --staging --staging-ref --production /chatops gitlab run feature delete rate_limiter_use_labkit_pipelines_create_enforce --dev --pre --staging --staging-ref --production /chatops gitlab run feature delete rate_limiter_use_labkit_notes_create --dev --pre --staging --staging-ref --production /chatops gitlab run feature delete rate_limiter_use_labkit_notes_create_enforce --dev --pre --staging --staging-ref --production /chatops gitlab run feature delete rate_limiter_use_labkit_search_rate_limit --dev --pre --staging --staging-ref --production /chatops gitlab run feature delete rate_limiter_use_labkit_search_rate_limit_enforce --dev --pre --staging --staging-ref --production /chatops gitlab run feature delete rate_limiter_use_labkit_users_get_by_id --dev --pre --staging --staging-ref --production /chatops gitlab run feature delete rate_limiter_use_labkit_users_get_by_id_enforce --dev --pre --staging --staging-ref --production /chatops gitlab run feature delete rate_limiter_use_labkit_user_sign_in --dev --pre --staging --staging-ref --production /chatops gitlab run feature delete rate_limiter_use_labkit_user_sign_in_enforce --dev --pre --staging --staging-ref --production /chatops gitlab run feature delete rate_limiter_use_labkit_cohort_2 --dev --pre --staging --staging-ref --production /chatops gitlab run feature delete rate_limiter_use_labkit_cohort_2_enforce --dev --pre --staging --staging-ref --production /chatops gitlab run feature delete rate_limiter_use_labkit_cohort_3 --dev --pre --staging --staging-ref --production /chatops gitlab run feature delete rate_limiter_use_labkit_cohort_3_enforce --dev --pre --staging --staging-ref --production /chatops gitlab run feature delete rate_limiter_use_labkit_cohort_4 --dev --pre --staging --staging-ref --production /chatops gitlab run feature delete rate_limiter_use_labkit_cohort_4_enforce --dev --pre --staging --staging-ref --production /chatops gitlab run feature delete rate_limiter_use_labkit_cohort_6 --dev --pre --staging --staging-ref --production /chatops gitlab run feature delete rate_limiter_use_labkit_cohort_6_enforce --dev --pre --staging --staging-ref --production ``` > The two cohort 5 flags (`rate_limiter_use_labkit_cohort_5`, `rate_limiter_use_labkit_cohort_5_enforce`) are intentionally omitted above. Add them to the cleanup MR and ChatOps run only after cohort 5 (gitlab-com/gl-infra/production-engineering#28812) is enforced and soaked. After ChatOps confirms the flags are deleted, close this issue (once cohort 5 is also cleaned up) and tick the remaining cleanup boxes on rollout issues gitlab-org/gitlab#598560 and gitlab-org/gitlab#599632. ## References - Cohort 1 rollout: gitlab-com/gl-infra/production-engineering#28803 - Cohort 2 rollout: gitlab-com/gl-infra/production-engineering#28809 - Cohort 3 rollout: gitlab-com/gl-infra/production-engineering#28810 - Cohort 4 rollout: gitlab-com/gl-infra/production-engineering#28811 - Cohort 5 rollout: gitlab-com/gl-infra/production-engineering#28812 - Cohort 6 rollout: gitlab-com/gl-infra/production-engineering#29077 - .com rollout issue (cohort 1+2): gitlab-org/gitlab#598560 - .com rollout issue (cohort 3): gitlab-org/gitlab#599632 - Migration MRs: gitlab-org/gitlab!233816 (cohort 1), gitlab-org/gitlab!234565 (cohort 2), gitlab-org/gitlab!235212 (cohort 3) - Late additions: gitlab-org/gitlab!239752 (`token_exchange`, cohort 6), gitlab-org/gitlab!239753 (`project_generate_new_export`, cohort 2) - Epic: gitlab-com/gl-infra&2021