## Problem `GET /api/:version/ai/duo_workflows/workflows/:id/checkpoints` (internal, used by Duo Workflow, feature category `duo_agent_platform`) allocates a lot of memory per request. The median request allocates about 109 MB. The largest allocate close to 1.5 GB. All of this traffic runs on the `ai-assisted` fleet. Code: `ee/lib/api/ai/duo_workflows/workflows_internal.rb`, lines 246-284 (the `get do` block), with the helpers `compress_checkpoint` (line 62) and `reconstruct_page!` (line 76). Entity: `ee/lib/api/entities/ai/duo_workflows/checkpoint.rb`. ## Data Last 24 hours, about 2,300 requests. | Metric | Value | |---|---| | Allocated per request | p50 109 MB, p90 323 MB, p99 749 MB, p99.9 1,445 MB, max 1,473 MB | | Requests over 256 MB / 512 MB / 1 GB | 352 / 74 / 9 | | Share of bytes that are malloc (large strings) | 89% | | Duration | p50 0.45 s, p99 3.9 s; largest requests 1.3 to 8.9 s | - In the last 6 hours, the 40 largest requests allocated 430 to 1,443 MB each, with only 1 to 4 million objects. - The largest requests are almost all `per_page=20&page=1&accept_compressed=true`. A few use `per_page=10`. - Typical small requests use `per_page=2` on a later page and allocate 15 to 100 MB. ## Where the memory likely goes This is a hypothesis from reading the code. It has not been measured. - One page loads up to `per_page` full checkpoints. On the incremental path, `reconstruct_page!` loads slim headers, rebuilds `channel_values` from blobs, and preloads checkpoint writes. - With `accept_compressed=true`, each checkpoint goes through `to_json`, then `Zlib::Deflate`, then `Base64.strict_encode64`. That creates several full-size string copies per checkpoint. - The whole page is then serialized into the response JSON. - So allocation likely scales with checkpoint size, times `per_page`, times the number of copies. ## Proposal Options to investigate: - Find out which client makes the `per_page=20&page=1` calls, and whether it needs 20 full checkpoints or only the latest one. - Lower the page size, or add a cap, for that caller. - Avoid building several full copies per checkpoint, for example by serializing once or by streaming. - Measure peak retained memory of one large request locally, using a large real-shaped workflow, before and after any change. ## Links - [Kibana: the largest requests to this endpoint, sorted by `json.mem_total_bytes`](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:'2026-09-23T23:00:00.000Z',to:'2026-09-24T23:00:00.000Z'))&_a=(columns:!('json.meta.caller_id','json.type','json.mem_total_bytes','json.mem_bytes','json.mem_objects','json.duration_s','json.cpu_s','json.status','json.path','json.params'),dataSource:(dataViewId:'7092c4e2-4eb5-46f2-8305-a7da2edad090',type:dataView),query:(language:kuery,query:'json.meta.caller_id.keyword%20:%20%22GET%20%2Fapi%2F:version%2Fai%2Fduo_workflows%2Fworkflows%2F:id%2Fcheckpoints%22'),sort:!(!('json.mem_total_bytes',desc)))) - [Grafana: request rate for this endpoint on ai-assisted](https://dashboards.gitlab.net/explore?schemaVersion=1&orgId=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22mimir-gitlab-gprd%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22sum%28rate%28gitlab_sli_rails_request_apdex_total%7Benv%3D%5C%22gprd%5C%22%2Ctype%3D%5C%22ai-assisted%5C%22%2Cendpoint_id%3D%5C%22GET%20%2Fapi%2F%3Aversion%2Fai%2Fduo_workflows%2Fworkflows%2F%3Aid%2Fcheckpoints%5C%22%7D%5B5m%5D%29%29%22%2C%22range%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22mimir-gitlab-gprd%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-7d%22%2C%22to%22%3A%22now%22%7D%7D%7D) - [Grafana: worst ai-assisted Puma container memory vs its limit, 7 days](https://dashboards.gitlab.net/explore?schemaVersion=1&orgId=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22mimir-gitlab-gprd%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22max%28container_memory_working_set_bytes%7Benv%3D%5C%22gprd%5C%22%2Cnamespace%3D%5C%22gitlab%5C%22%2Ccontainer%3D%5C%22webservice%5C%22%2Cpod%3D~%5C%22gitlab-webservice-ai-assisted.%2A%5C%22%7D%29%22%2C%22range%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22mimir-gitlab-gprd%22%7D%7D%2C%7B%22refId%22%3A%22B%22%2C%22expr%22%3A%22max%28kube_pod_container_resource_limits%7Benv%3D%5C%22gprd%5C%22%2Cnamespace%3D%5C%22gitlab%5C%22%2Ccontainer%3D%5C%22webservice%5C%22%2Cpod%3D~%5C%22gitlab-webservice-ai-assisted.%2A%5C%22%2Cresource%3D%5C%22memory%5C%22%7D%29%22%2C%22range%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22mimir-gitlab-gprd%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-7d%22%2C%22to%22%3A%22now%22%7D%7D%7D) - [Grafana: ai-assisted Puma container restarts per hour, 7 days](https://dashboards.gitlab.net/explore?schemaVersion=1&orgId=1&panes=%7B%22a%22%3A%7B%22datasource%22%3A%22mimir-gitlab-gprd%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22sum%28increase%28kube_pod_container_status_restarts_total%7Benv%3D%5C%22gprd%5C%22%2Cnamespace%3D%5C%22gitlab%5C%22%2Ccontainer%3D%5C%22webservice%5C%22%2Cpod%3D~%5C%22gitlab-webservice-ai-assisted.%2A%5C%22%7D%5B1h%5D%29%29%22%2C%22range%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22mimir-gitlab-gprd%22%7D%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-7d%22%2C%22to%22%3A%22now%22%7D%7D%7D) The Kibana links cover 2026-09-23 23:00 to 2026-09-24 23:00 UTC and stop working once those logs expire. ## Context - `mem_total_bytes` in the Rails API logs is the cumulative number of bytes allocated during one request (malloc bytes plus object slots, see `lib/gitlab/memory/instrumentation.rb`). It is not peak memory. Memory freed during the request is still counted. How much is live at once is not logged. - Large allocations still raise Puma worker memory and GC pressure. In gprd, the `ai-assisted` webservice pods (4 Puma workers each) are OOM-killed several times a day, with 45 container restarts in 7 days. - We have not tied any specific request to a specific OOM kill. Rails does not log requests that are in flight when a container is killed. - Pod memory is being raised in https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/5853. Capacity context: https://gitlab.com/gitlab-com/gl-infra/capacity-planning-trackers/gitlab-com/-/work_items/2668 - Data source: gprd Rails API logs in Kibana (`pubsub-rails-inf-gprd*`), last 24 hours, as of 2026-09-25. --- _Data gathered and drafted with AI assistance; reviewed by me._