### Problem `Security::ScanProfileStatus::UpdateWorker` has no `concurrency_limit` today. During the burst investigated in #630653, it ran unthrottled and produced the peak described there: 308,354 sec-DB write statements in 10 minutes, about 514 per second, versus 13,276 for the next largest writer. ### Why a concurrency limit fits here WAL saturation is a rate metric. The alert that fired is a z-score on `rate(pg_xlog_position_bytes[5m])`. A concurrency limit caps how many executions of this worker run at once, which caps write rate directly, the same quantity the alert measures. This is a different situation from the identifier table work elsewhere in this epic. There, the writes were no-ops and the fix was to stop issuing them. Here the writes are real: `last_scan_at`, `build_id` and the consecutive counters change on every run [`ee/app/services/security/scan_profile_status/update_service.rb:38-41`](https://gitlab.com/gitlab-org/gitlab/-/blob/bef9c8a49773bf492c29ca650abab015b1e4f836/ee/app/services/security/scan_profile_status/update_service.rb#L38-41). We cannot eliminate them, so pacing is the lever we have. ### What it will and will not do `concurrency_limit` defers excess jobs into a Redis buffer, it does not drop them. Total work is unchanged. The backlog drains through `ConcurrencyLimit::ResumeWorker`, which tops execution back up to `limit - current` on each run. We are trading a sharp peak for a longer, lower plateau. This buys quiet alerts, not less WAL. ### Sizing Peak was 308,354 writes over 600 seconds, 514 writes per second. A successful fast-path job takes about 0.10 seconds, so each concurrency slot delivers roughly 10 writes per second. Sustaining the peak needed roughly 50 slots of fast-path work. True concurrency during the burst was higher than that, because a contended job spends about 4.5 seconds in a slot before it writes anything, roughly 45 times as long as a fast-path job occupying the same slot. Rough scale: - around 50: roughly a no-op limit - 20 to 25: roughly halves the peak write rate - around 15: roughly a third of the peak write rate Existing workers under `ee/app/workers/security/` set `concurrency_limit` between 10 and 300, so 20 to 30 is within house norms. The final number should be confirmed against job duration at rollout time rather than fixed here. ### A consequence to decide before shipping Lowering concurrency reduces per-project lease collisions. Fewer concurrent jobs against the same project means fewer 4.5 second lease losers and more 0.10 second successful writers, which means less Sidekiq capacity burned on the 4 second lease sleep. That part is good. But those recovered jobs still write. So total write volume can go up even as the peak rate comes down. Expect the spike to flatten; do not expect total WAL to fall. ### Backlog and staleness Deferred jobs queue in Redis. On the day of the incident there were about 3.9 million executions in a 14 hour window, so at a limit low enough to matter the backlog would run into the hundreds of thousands and take hours to drain. Scan profile badges on the Security Configuration page would read stale for that period. The badge is advisory and nothing gates on it, so this is likely acceptable, but it is a product call worth making explicitly rather than letting it fall out of the number we pick. ### Implementation notes Declare `concurrency_limit -> { N }` on the worker alongside the other worker attributes [`ee/app/workers/security/scan_profile_status/update_worker.rb:16`](https://gitlab.com/gitlab-org/gitlab/-/blob/bef9c8a49773bf492c29ca650abab015b1e4f836/ee/app/workers/security/scan_profile_status/update_worker.rb#L16). The value is tunable at runtime without a deploy: `LimitManager` stores the current limit in Redis with a TTL, and `ConcurrencyLimitService.set_current_limit!` overrides it. Precedent: !256757 added a concurrency limit to `Security::StoreSecurityReportsByProjectWorker` for the same reason. ### Acceptance criteria - [ ] `concurrency_limit` declared on the worker - [ ] Peak per-10-minute sec write count for this worker visibly reduced on the Kibana panel used in #630653 - [ ] Backlog drains within an acceptable window and does not grow unbounded across a normal day - [ ] No increase in job failures for this worker Related code: lease constants [`ee/app/workers/security/scan_profile_status/update_worker.rb:23-26`](https://gitlab.com/gitlab-org/gitlab/-/blob/bef9c8a49773bf492c29ca650abab015b1e4f836/ee/app/workers/security/scan_profile_status/update_worker.rb#L23-26), the `in_lock` call [`ee/app/workers/security/scan_profile_status/update_worker.rb:38-42`](https://gitlab.com/gitlab-org/gitlab/-/blob/bef9c8a49773bf492c29ca650abab015b1e4f836/ee/app/workers/security/scan_profile_status/update_worker.rb#L38-42), the upsert [`ee/app/services/security/scan_profile_status/update_service.rb:38-41`](https://gitlab.com/gitlab-org/gitlab/-/blob/bef9c8a49773bf492c29ca650abab015b1e4f836/ee/app/services/security/scan_profile_status/update_service.rb#L38-41). Follow-up to #630653, part of &23596.