## Summary Roll out [the feature](https://gitlab.com/gitlab-org/gitlab/-/work_items/300855) currently behind the `blob_permalink_embed` feature flag. - DRI: @kivikakk - Team Slack channel: `#g_planner-intelligence` **Note** there's another very similarly-named rollout issue (https://gitlab.com/gitlab-org/gitlab/-/work_items/604827+) created for an unmerged contributor MR (https://gitlab.com/gitlab-org/gitlab/-/merge_requests/240543+). This issue relates to the actually merged FF. ## What could go wrong? In terms of blast radius: * This is a post-processing filter, so when enabled, changes are seen (for the enabled actors) instantly. * Likewise, disabling is instant: no cache concerns. * The trigger condition is pretttty narrow (standalone link as the only content of its paragraph), fully-qualified permalink, so unlikely to occur by accident! * FF actor is the project where embeds are to be enabled, rather than the project whose code is being embedded. * i.e. you enable _access to the feature_ for a project, rather than enabling the feature to be _used on_ projects. * Permissions checks are always still relevant: if **you** can't see the target, the embed doesn't appear for you. In terms of perf (where a 'field' is any single rendered piece of text; an MR description, a single note, etc.): * We limit 100 "candidates" per field to try, 10 embeds to actually fetch and render per field. * i.e. if some of the candidates turn out to be over limit, or binaries, or in LFS, we keep going. * The embeds are fetched from Gitaly in one batched call per field. * We limit 100 lines max to display in any embed, 5000 lines to highlight to achieve that. If either limit is broken, we don't even try. * When we "keep going" or "don't even try", that result is cached the same as a successful result. Transient errors aren't. * Cache is in Redis. * This means if you have 30 notes on a page all with embeds, you can have 30 Gitaly calls on a cold cache. * We have per-request render limits and `PipelineTimingCheck` as our usual backstops. In terms of security: * Existing permissions checks are respected; you can never see something you shouldn't. * The design also avoids ever storing code in unredacted persistent caches, so we don't add a path for disclosures of unredacted content to expose code. (This is a key difference to !240543.) * Disabling "Include diff previews" under "Email notifications" for a project stops embeds in that project **and** stops that project from being embedded. * See https://gitlab.com/gitlab-org/gitlab/-/merge_requests/251058+. Arguably we could remove the former of those conditions, but we're erring on the side of non-disclosure. In terms of completeness: * These currently render poorly in emails. Will be addressed by !251072+. ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to `#g_planner-intelligence`. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set blob_permalink_embed 50 --actors --dev --pre --staging --staging-ref /chatops gitlab run feature set blob_permalink_embed true --dev --pre --staging --staging-ref ``` **Production** — percentage rollout (wait ≥15 min between steps, watch dashboards): ``` /chatops gitlab run feature set blob_permalink_embed <percentage> --actors ``` Or target specific actors instead: ``` /chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss blob_permalink_embed true /chatops gitlab run feature set --group=gitlab-org,gitlab-com blob_permalink_embed true /chatops gitlab run feature set --user=kivikakk blob_permalink_embed true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [Docs + version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any - [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Track it here, or open a follow-up [Feature Flag Cleanup issue](https://gitlab.com/gitlab-org/gitlab/-/work_items/new?description_template=Feature%20Flag%20Cleanup). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run release check <merge-request-url> <milestone> /chatops gitlab run feature delete blob_permalink_embed --dev --pre --staging --staging-ref --production ``` ## Rollback ``` /chatops gitlab run feature set blob_permalink_embed false # production /chatops gitlab run feature set blob_permalink_embed false --dev --pre --staging --staging-ref # non-production /chatops gitlab run feature delete blob_permalink_embed --dev --pre --staging --staging-ref --production # remove entirely ```