User interaction with Merge Request causes significant spike in web requests
Summary
A Dedicated customer experienced workhorse service degradation INC-5133, attributed to a large spike in web requests from an individual user. Support confirmed with the user that they were simply interacting with a single Merge Request via the browser (no automation used). When investigating the user's requests over the relevant 6-hour period, we found that the majority of the 424,362 web requests were to the following paths:
-
<PROJECT_PATH>/-/merge_requests/104/widget.json(49.81%) -
/api/graphql(31.6%) -
<PROJECT_PATH>/-/merge_requests/104/cached_widget.json(18.46%)
This was a surprising/significant number of requests to have spawned from normal user interaction with a Merge Request. The majority of these requests (99.61%) were of meta.feature_category: code_review_workflow. There was not one correlation ID tied to the majority of requests.
Impact
This significant increase in web requests from the user's Merge Request interaction caused significant degradation for the workhorse SLI apdex (meaning a large portion of workhorse requests were taking longer than the acceptable threshold to complete), leading to an incident INC-5133:
Recommendation
Investigate why this user's activity spawned such a large spike in web requests, causing workhorse performance degradation on the Dedicated tenant, and remediate.