[FF] async_write_diffs_cache_on_mr_create -- Defer MR diffs cache write to a worker on creation

Summary

Roll out the feature currently behind the async_write_diffs_cache_on_mr_create feature flag.

The flag defers the merge request diffs highlight cache write in MergeRequests::AfterCreateService to the new MergeRequests::WriteDiffsCacheWorker, moving the CPU-bound syntax highlighting (~11s at P99 on large merge requests) off the new-MR preparation critical path. Introduced in !243117 (merged).

  • DRI: @marc_shaw
  • Team Slack channel: #g_code_review

Note

Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. Feature flag controls · Feature flag lifecycle

What could go wrong?

The cache is display-only, so the blast radius is small:

  • If the worker is delayed or fails, the first diff render hits a cold cache and warms it on demand (Gitlab::Diff::HighlightCache#write_if_empty); subsequent renders are warm. This means a slightly slower first render, never missing or incorrect data.
  • The cache key is content-addressed on the immutable merge request diff, so a concurrent worker write and on-render write are idempotent and cannot corrupt each other.
  • Nothing on the merge request readiness, mergeability, pipeline, approval, or webhook paths reads this cache.

Dashboards to watch on https://dashboards.gitlab.net:

  • MergeRequests::WriteDiffsCacheWorker Sidekiq execution/queue latency (urgency :low, boundary :cpu) — watch for Sidekiq CPU saturation during merge-request-creation bursts (for example, large imports).
  • MergeRequests::AfterCreateService / NewMergeRequestWorker duration — expect the creation path to get faster as the write moves off it.

Rollout

Run all production /chatops in #production and cross-post the results to #g_code_review. Background: incremental rollout process, feature actors.

The flag actor is the merge request target project.

Non-production

/chatops gitlab run feature set async_write_diffs_cache_on_mr_create 50 --actors --dev --pre --staging --staging-ref
/chatops gitlab run feature set async_write_diffs_cache_on_mr_create true --dev --pre --staging --staging-ref

Production — percentage rollout (wait ≥15 min between steps, watch dashboards):

/chatops gitlab run feature set async_write_diffs_cache_on_mr_create <percentage> --actors

Or target specific actors instead:

/chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss async_write_diffs_cache_on_mr_create true
/chatops gitlab run feature set --group=gitlab-org,gitlab-com async_write_diffs_cache_on_mr_create true
/chatops gitlab run feature set --user=marc_shaw async_write_diffs_cache_on_mr_create true

Before global rollout

Confirm the relevant gotchas before going to 100% — see enabling a feature for GitLab.com:

Cleanup

Remove the flag once deemed stable — see cleaning up. Track it here, or open a follow-up Feature Flag Cleanup issue. Remove the flag and its YAML definition from the codebase, then:

/chatops gitlab run release check https://gitlab.com/gitlab-org/gitlab/-/merge_requests/243117 <milestone>
/chatops gitlab run feature delete async_write_diffs_cache_on_mr_create --dev --pre --staging --staging-ref --production

Rollback

/chatops gitlab run feature set async_write_diffs_cache_on_mr_create false                                         # production
/chatops gitlab run feature set async_write_diffs_cache_on_mr_create false --dev --pre --staging --staging-ref     # non-production
/chatops gitlab run feature delete async_write_diffs_cache_on_mr_create --dev --pre --staging --staging-ref --production  # remove entirely