## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/591927) currently behind the `skip_oversized_diff_blobs` feature flag. Introduced in https://gitlab.com/gitlab-org/gitlab/-/merge_requests/258929. - DRI: @francoisrose (engineering manager, ~"group::code review"), @vyaklushin (merging maintainer) - Team Slack channel: `#g_code-review` > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## What could go wrong? - The flag changes how merge request diffs load on the Changes tab (`diffs_batch`, `diffs_metadata`, rapid diffs stream). That code runs for every merge request diff view. An error there affects the whole Changes tab. - With the flag on, a page with binary diffs, stored too-large patches or renames without a patch makes one extra metadata-only Gitaly `GetBlobs` call (limit 0). Each call covers up to 250 blob references, at most two per diff. Larger sets take one call per 250 references, one after another. The diff files reuse the result, so no second call follows. Locally the call took 28 ms for a 300 MB blob. Pages without such diffs make no extra call. Collapsed text diffs and renames with a text patch are not checked. - Files whose old or new blob is above 50 MB show as "too large" instead of "binary" or "not diffable". No content is lost. Users can still view the blob. - Turning the flag off does not take effect for pages that are already cached. The `diffs_batch` response cache and the highlight cache do not include the flag in their keys. Those pages keep showing the file as too large until the cache expires. - What to watch: the Gitaly `GetBlobs` call rate and latency. Also watch the error rate of `Projects::MergeRequests::DiffsController#diffs_batch` and `#diffs_metadata`. ## Events - [Exceptions with skip_oversized_diff_blobs:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.exception.feature_flag_states.keyword:%22skip_oversized_diff_blobs:1%22'),columns:!(json.exception.class,json.exception.message,json.extra.workflow_id,json.meta.caller_id),sort:!(!(json.time,desc)))) - [Events with skip_oversized_diff_blobs:1](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:now-1d,to:now))&_a=(index:'7092c4e2-4eb5-46f2-8305-a7da2edad090',query:(language:kuery,query:'json.feature_flag_states:%22skip_oversized_diff_blobs:1%22'),columns:!(json.meta.caller_id,json.feature_flag_states),sort:!(!(json.time,desc)))) - [Error rate and other graphs by modifying the examples in the Visualization Library](https://log.gprd.gitlab.net/app/visualize#/?_g=h@358d019&s=FF%20Observability) Feature Flag events are only logged by default for feature flags marked for the current or future milestones. To enable while the feature flag is active, see https://docs.gitlab.com/development/feature_flags/#logging ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results in this issue. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set skip_oversized_diff_blobs 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set skip_oversized_diff_blobs true --dev --pre --staging --staging-ref ``` **Production** — percentage rollout (wait ≥15 min between steps, watch dashboards): ``` /chatops gitlab run feature set skip_oversized_diff_blobs <percentage> --actors ``` Or target specific projects instead. The flag is checked with the project as actor, so only `--project` gates apply. `--group` and `--user` gates have no effect. ``` /chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss skip_oversized_diff_blobs true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete skip_oversized_diff_blobs --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set skip_oversized_diff_blobs false # production /chatops gitlab run feature set skip_oversized_diff_blobs false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete skip_oversized_diff_blobs --dev --pre --staging --staging-ref --production # remove entirely ```