Degraded Time to First Byte performance for list merge requests API
Summary
Project Merge Requests API endpoint (api_v4_projects_merge_requests) and Group Merge Requests API endpoint degraded in performance since 2025-12-15.
-
api_v4_projects_merge_requestsTime to First Byte (TTFB) p90 has increased from ~300ms to 2+ seconds under load -
api_v4_groups_merge_requestsTTFB has increased from < 3s to 5s+
-
10k Grafana dashboards (Grafana dashboard login credentials are stored in
Engineering1Password vault underPerformance Grafana dashboard credentials)
Note that other MR endpoints do not have such degradation as list MR endpoints.
The diff between first failure and last success run is 6fd0e12d...bcc16321 indicates that MR Enable granular PATs for MR diffs (!215791 - merged) made changes to lib/api/merge_requests so marking as potentially related, however it needs to be confirmed and investigated.
Steps to reproduce
- Set up a GitLab instance
- Run the GitLab Performance Tool pipeline against the endpoint
api_v4_projects_merge_requests - Monitor TTFB p90 metric
Example Project
Testing was performed on the 10k Reference Architecture environment using the gitlabhq project.
What is the current bug behavior?
TTFB p90 is significantly degraded
What is the expected correct behavior?
TTFB p90 results restored to previous values
Possible fixes
It's possible to bisect commits between 6fd0e12d802 and bcc16321fd8. There is an option to use GDK to identify the exact culprit
