PUT /api/:version/projects/:id/issues/:issue_iid memory utilization issues
Update: I see that we're not include new notes, just changing the labels on the issue. This has happened so many times that the issue has a very long list of events that's causing memory issues and timeouts.
Perhaps this should be an Application Limits issue, in that if an issue has more than X (say 1000) events, we just stop appending, or truncate old events?
1% of PUT /api/:version/projects/:id/issues/:issue_iid
endpoint requests use more than 400MB of memory.
Likewise, requests to GET /api/:version/projects/:id/issues/:issue_iid
use almost as much memory.
As reported in Mechanical Sympathy report
https://gitlab.slack.com/archives/CM5EQH125/p1617961434017700
It's worth noting that all the worst offenders, over the past 24 hours, using well in excess of 1GiB per request, are for issues in the gitlab-org/quality/testcases
repository (see Kibana search link above for examples).
Useragent is Gitlab Ruby Gem 4.16.1
Looking at one of the issues that is consistently using more than 1GB of memory, https://gitlab.com/gitlab-org/quality/testcases/-/issues/394
, the issue appears to have so many notes that it is unable to load. I'm assuming that every success or failure of this QA job, over an extended period, is recorded in this issue. If this is the case, I wonder whether we should consider not doing this, as storage of huge numbers of notes is not what GitLab issues are intended to be used for.