Limit cross-reference checks per save and per user

What does this MR do and why?

When a description, comment or commit is saved, GitLab checks every reference in it and adds a "mentioned in" note to each referenced item that doesn't have one yet. This MR bounds that work per save and per user:

  • Per save: a save checks at most 150 references. Past that, references are kept in this order: work items and issues, then merge requests, then commits, then epics.
  • Per user: each reference checked counts against a new cross_reference_checks rate limit, 800 a minute for each user. A save is charged at most the limit, so one large save still goes through in a fresh minute. The limit is the cross_reference_checks_limit application setting, labeled beta in the docs. Internal users, GitLab.com integration users, and service accounts with composite identity are counted for each project or group instead of across all of them. Project and group access token users have a single limit, like other users.

The save always succeeds. Over the cap, the references past 150 get no "mentioned in" note. Over the rate limit, the save adds no "mentioned in" notes. Saves over the cap, and saves cut by the rate limit, write a Cross-reference limits exceeded log line. Every caller runs after the save or in a background job, so returning an error there would skip the work that follows the save.

The change is in Mentionable#create_cross_references!, which every save path that adds "mentioned in" notes goes through. Importers write their notes directly and are not affected. Commits keep the per-save cap but skip the rate limit.

The rate limit is defined with staged_rule, so it rolls out in log mode first. One feature flag, enforce_cross_reference_limits, disabled by default:

  • Flag disabled: the rate limit counts each save of a description or comment that has references, and blocks nothing. This adds one rate limiter check to each of those saves. A save over the limit shows as applimiter_cross_reference_checks:limit_cross_reference_checks_by_user_log:log in rate_limit_state on the request or job log line. The cap is not applied, but a save over it writes the log line.
  • Flag enabled: the cap and the rate limit apply.

The flag is checked for the user who saves.

Values chosen from GitLab.com data, details in the confidential issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/638274.

References

Follow-up

How to set up and validate locally

Click to expand
  1. Enable the flag in the Rails console:

    Feature.enable(:enforce_cross_reference_limits)
  2. Lower the rate limit with the application settings API, as an administrator:

    curl --request PUT --header "PRIVATE-TOKEN: <admin token>" "http://gdk.test:3000/api/v4/application/settings?cross_reference_checks_limit=3"
  3. Wait about a minute. Application settings and feature flags are cached in each process.

  4. Create two issues that each reference two other issues. The first gets its "mentioned in" notes on both referenced issues. The second does not, and log/application_json.log has a Cross-reference limits exceeded line with throttled: true.

    Referenced by the first issue Referenced by the second issue
    target_15 target_17
  5. Disable enforce_cross_reference_limits, wait about a minute, and repeat. The notes are added, and the NewIssueWorker line in log/sidekiq.log has applimiter_cross_reference_checks:limit_cross_reference_checks_by_user_log:log in rate_limit_state.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Agnes Slota

Merge request reports

Loading
Loading