<!-- Title: [FF] `limit_phone_verification_transactions_by_dial_code` -- require credit card verification once a dial code exceeds its daily SMS limit --> ## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/609536) currently behind the `limit_phone_verification_transactions_by_dial_code` feature flag, introduced in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/257995. With the flag on, GitLab counts the verification SMS messages sent to each dial code over a rolling 24 hours. Once a dial code reaches its limit, the next user who enters a phone number with that dial code gets no SMS. Instead they must verify a credit card first. Users who have already verified a card, and users already marked as assumed high risk, are not affected. - Default limit: 50 per dial code per day (`dial_code_phone_verification_transactions_daily_limit`) - Per-dial-code overrides: `{ "1" => 800 }` (`dial_code_phone_verification_transactions_daily_limit_overrides`) - A limit of `0` turns the limit off, either globally or for a single dial code. - DRI: @jayswain - Team Slack channel: `#<slack-channel-of-dri-team>` > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? The flag affects every user on a dial code once that code passes its daily limit. It is not limited to users who look suspicious. - **The default limit is low for busy dial codes.** Only `+1` has an override. Any other dial code that normally sends more than 50 verification SMS messages a day will start requiring a credit card from every new user after the 50th, every day. Check real daily volume per dial code before enabling, and add overrides for the busy ones. - **Counts start at zero when the flag is turned on.** Sends are only counted while the flag is on, so we cannot measure the counts in advance. For the first 24 hours after enabling, limits trip as the day's traffic builds up. - **The limits can't be changed from the Admin area or the API.** They are only available through the Rails console, so changing a limit on GitLab.com needs a console change request. Prepare the override values before enabling. - **Turning the flag off does not undo the assumed high risk marking.** Affected users are marked assumed high risk, which is a stored custom attribute. They keep being asked for a credit card after a rollback, so those attributes have to be cleared separately. - **Users with no card cannot finish phone verification** on a dial code that is over its limit until the rolling window clears. Watch identity verification completion rates by country, SMS spend, and the limit-exceeded log event below. ## Events - [Exceptions with limit_phone_verification_transactions_by_dial_code:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.exception.feature_flag_states.keyword:%22limit_phone_verification_transactions_by_dial_code:1%22'),columns:!(json.exception.class,json.exception.message,json.extra.workflow_id,json.meta.caller_id),sort:!(!(json.time,desc)))) - [Events with limit_phone_verification_transactions_by_dial_code:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.feature_flag_states:%22limit_phone_verification_transactions_by_dial_code:1%22'),columns:!(json.meta.caller_id,json.feature_flag_states),sort:!(!(json.time,desc)))) - [Error rate and other graphs by modifying the examples in the Visualization Library](https://log.gprd.gitlab.net/app/visualize#/?_g=h@358d019&s=FF%20Observability) - Dial codes hitting their limit, in the `pubsub-rails-inf-gprd*` data view: ``` json.message: "IdentityVerification::Phone" and json.exceeded_limit: "dial_code_phone_verification_transactions_limit" ``` Break down by `json.international_dial_code`. Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#<slack-channel-of-dri-team>`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). > [!warning] > The flag is checked with the `:instance` actor, because the limit is counted per dial code across all users. Percentage-of-actors rollouts and project, group, or user targeting have no effect: the flag is either on for everyone or off. Use the per-dial-code limits to control impact instead. **Before enabling in production** - [ ] Review daily verification SMS volume per dial code on GitLab.com. - [ ] Open a change request to set `dial_code_phone_verification_transactions_daily_limit_overrides` for every dial code that normally sends more than the default limit. **Non-production** ``` /chatops gitlab run feature set limit_phone_verification_transactions_by_dial_code true --dev --pre --staging --staging-ref ``` **Production** ``` /chatops gitlab run feature set limit_phone_verification_transactions_by_dial_code true ``` If a legitimate dial code trips its limit, raise its override or set it to `0` rather than turning off the whole flag. ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete limit_phone_verification_transactions_by_dial_code --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set limit_phone_verification_transactions_by_dial_code false # production /chatops gitlab run feature set limit_phone_verification_transactions_by_dial_code false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete limit_phone_verification_transactions_by_dial_code --dev --pre --staging --staging-ref --production # remove entirely ``` After a rollback, users marked assumed high risk during the rollout still have to verify a credit card. Clear those custom attributes separately if needed.