## Problem `POST /api/:version/ai/duo_workflows/code_review/add_comments` (feature category `duo_code_review`) has a long allocation tail. Most requests allocate under 30 MB, but some allocate hundreds of MB, and one allocated 1.7 GB. The request body is small, so the memory does not come from the input. All of this traffic runs on the `ai-assisted` fleet. Code: `ee/lib/api/ai/duo_workflows/code_review.rb` lines 65-87. Service: `ee/app/services/ai/duo_workflows/code_review/process_review_output_service.rb`. ## Data Last 24 hours, about 13,500 requests (201 responses, plus 93 x 400 and 45 x 500). | Metric | Value | |---|---| | Allocated per request | p50 18 MB, p90 29 MB, p99 94 MB, p99.9 397 MB, max 1,763 MB | | Requests over 256 MB / 512 MB / 1 GB | 33 / 7 / 1 | | Share of bytes that are malloc | 80% | | Request body size | p50 2 KB, p99 14 KB | | Duration | p50 0.44 s, p99 3.9 s | ## Where the memory likely goes This is a hypothesis from reading the code. It has not been measured. - The synchronous request only validates the input and then enqueues `PublishCommentsWorker`. - `valid?` calls `ai_reviewable_diff_files` in `ee/app/services/ai/duo_workflows/code_review/file_exclusion.rb`. - That calls `MergeRequest#ai_reviewable_diff_files` in `ee/app/models/ee/merge_request.rb` (around line 811). - That loads `diffs(expanded: true).diff_files` for the whole merge request, fully expanded, only to check that the list is not empty. - On large merge requests this could explain the tail. The worker probably loads the diffs again later. ## Proposal Options to investigate: - Confirm the hypothesis with a local profile on a large merge request. - Make the emptiness check cheaper, for example by stopping at the first reviewable file or by not using `expanded: true` for this check. Leave the full expanded load to the worker. - Verify that the 400 `nothing_to_review` behavior stays the same. ## Links - [Kibana: the largest requests to this endpoint, sorted by `json.mem_total_bytes`](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:'2026-09-23T23:00:00.000Z',to:'2026-09-24T23:00:00.000Z'))&_a=(columns:!('json.meta.caller_id','json.type','json.mem_total_bytes','json.mem_bytes','json.mem_objects','json.duration_s','json.cpu_s','json.status','json.path'),dataSource:(dataViewId:'7092c4e2-4eb5-46f2-8305-a7da2edad090',type:dataView),query:(language:kuery,query:'json.meta.caller_id.keyword%20:%20%22POST%20%2Fapi%2F:version%2Fai%2Fduo_workflows%2Fcode_review%2Fadd_comments%22'),sort:!(!('json.mem_total_bytes',desc)))) - [Grafana: request rate for this endpoint on ai-assisted](https://dashboards.gitlab.net/explore?schemaVersion=1&orgId=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22mimir-gitlab-gprd%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22sum%28rate%28gitlab_sli_rails_request_apdex_total%7Benv%3D%5C%22gprd%5C%22%2Ctype%3D%5C%22ai-assisted%5C%22%2Cendpoint_id%3D%5C%22POST%20%2Fapi%2F%3Aversion%2Fai%2Fduo_workflows%2Fcode_review%2Fadd_comments%5C%22%7D%5B5m%5D%29%29%22%2C%22range%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22mimir-gitlab-gprd%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-7d%22%2C%22to%22%3A%22now%22%7D%7D%7D) - [Grafana: worst ai-assisted Puma container memory vs its limit, 7 days](https://dashboards.gitlab.net/explore?schemaVersion=1&orgId=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22mimir-gitlab-gprd%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22max%28container_memory_working_set_bytes%7Benv%3D%5C%22gprd%5C%22%2Cnamespace%3D%5C%22gitlab%5C%22%2Ccontainer%3D%5C%22webservice%5C%22%2Cpod%3D~%5C%22gitlab-webservice-ai-assisted.%2A%5C%22%7D%29%22%2C%22range%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22mimir-gitlab-gprd%22%7D%7D%2C%7B%22refId%22%3A%22B%22%2C%22expr%22%3A%22max%28kube_pod_container_resource_limits%7Benv%3D%5C%22gprd%5C%22%2Cnamespace%3D%5C%22gitlab%5C%22%2Ccontainer%3D%5C%22webservice%5C%22%2Cpod%3D~%5C%22gitlab-webservice-ai-assisted.%2A%5C%22%2Cresource%3D%5C%22memory%5C%22%7D%29%22%2C%22range%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22mimir-gitlab-gprd%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-7d%22%2C%22to%22%3A%22now%22%7D%7D%7D) - [Grafana: ai-assisted Puma container restarts per hour, 7 days](https://dashboards.gitlab.net/explore?schemaVersion=1&orgId=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22mimir-gitlab-gprd%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22sum%28increase%28kube_pod_container_status_restarts_total%7Benv%3D%5C%22gprd%5C%22%2Cnamespace%3D%5C%22gitlab%5C%22%2Ccontainer%3D%5C%22webservice%5C%22%2Cpod%3D~%5C%22gitlab-webservice-ai-assisted.%2A%5C%22%7D%5B1h%5D%29%29%22%2C%22range%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22mimir-gitlab-gprd%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-7d%22%2C%22to%22%3A%22now%22%7D%7D%7D) The Kibana links cover 2026-09-23 23:00 to 2026-09-24 23:00 UTC and stop working once those logs expire. ## Context - `mem_total_bytes` in the Rails API logs is the cumulative number of bytes allocated during one request (malloc bytes plus object slots, see `lib/gitlab/memory/instrumentation.rb`). It is not peak memory. Memory freed during the request is still counted. How much is live at once is not logged. - Large allocations still raise Puma worker memory and GC pressure. In gprd, the `ai-assisted` webservice pods (4 Puma workers each) are OOM-killed several times a day, with 45 container restarts in 7 days. - We have not tied any specific request to a specific OOM kill. Rails does not log requests that are in flight when a container is killed. - Pod memory is being raised in https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/5853. Capacity context: https://gitlab.com/gitlab-com/gl-infra/capacity-planning-trackers/gitlab-com/-/work_items/2668 - Data source: gprd Rails API logs in Kibana (`pubsub-rails-inf-gprd*`), last 24 hours, as of 2026-09-25. --- _Data gathered and drafted with AI assistance; reviewed by me._