Reduce urgency of Settings Sync API endpoint
What does this MR do and why?
This Merge Request lowers the urgency of the Settings Sync API endpoint from default to low so the threshold used to measure the apdex metric for this endpoint increases from 1s to 5s . In other words, it ensures that requests that take longer than 1 second to complete but less than 5 seconds won't affect the IDE group error budget.
Why
The Settings Sync API is used by the Web IDE to persist installed extensions and user settings. When we enabled the Settings Sync functionality in GitLab.com, we observed an spike in the number of slow requests in the error budget dashboard, but we couldn't identify the root cause as explained in #443637 (comment 1805993826). We will continue the investigation, but in the meantime, we are also considering the nature of the apdex metric and the Settings Sync endpoint.
The apdex metric measures the performance of an application as perceived by the user. The goal of this metric is measuring what matters, therefore, an API endpoint that doesn't impact the user perception of performance shouldn't impact the apdex metric. The Settings Sync endpoint satisfies this description.
How does Settings Sync work?
When the user changes a setting in the Web IDE or installs an extension, the Web IDE calls the Settings Sync endpoints to save these customizations. This operation happens in the background as demonstrated in this video, and the user triggers the operation indirectly. The user doesn't have to wait until the Settings Sync API responds to continue with their workflow. Based on the behavior of this feature, we assume that this endpoint satisfies the criteria to be a low-urgency endpoint.
Performance monitoring dashboards
- The following dashboard shows the performance of the 99.95th percentile of Settings Sync API requests: https://log.gprd.gitlab.net/app/dashboards#/view/b838eef0-dc99-11ee-9f58-4f945bb0e111?_g=h@d01ba18
- You can see the slow requests for the first week of March when the client-side feature flag for this feature was activated in https://log.gprd.gitlab.net/app/r/s/5CiZ3
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Related to #443637 (closed)