Gitlab for Jira app improve initial sync: Sync more merge requests instead of just last 400
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
About
When a new group is connected to Jira, we sync the latest 400 merge requests that have Jira key-like titles or descriptions, for each project in the group.
This syncs a limited amount of historical data.
Problem
Customers would like more historical MR data to be sync'd.
Technical considerations
The limit of 400 originated here !43880 (comment 425268599).
Impact on GitLab
As the sync happens for each project in the group we need to consider the impact of what happens when a group with thousands of projects, which have thousands of MRs, could be on services.
The past 7 days of db_duration_s (time spent in the database) for JiraConnect::SyncProjectWorker can be seen here: https://log.gprd.gitlab.net/app/r/s/sn9y9 split by 99th, 95th and 50th percentiles for the current limit of 400. At time of writing March 2025 the chart is:
99 percentile is around 1.5s-3.5s time spent in database, and 95th percentile as around 1s, and this is happening on the replicas.
More requests to Jira
The limit also matches the Jira API limit of how many pull requests you can include in the POST at a time - see the Jira docs (Request body > repositories > show child properties > pullRequests). This means to support a higher number of MRs, we'd need to batch the requests to Jira and make more than 1.
There are Jira API rate limits which we might tip towards.
Jira API endpoint limit of 400
As noted in MR that chose the limit !43880 (comment 425268599) the Jira API itself has a limit of 400 MRs you can include in the POST - see the Jira docs for the endpoint (Request body > repositories > show child properties > pullRequests).
This means in order to sync > 400 MRs we need to batch the API requests to Jira, sending 400 in each request.
Technical proposal
Given that the time spent in the database within the worker that syncs the MRs https://log.gprd.gitlab.net/app/r/s/sn9y9 shows the 99 percentile as around 1.5s-3.5s, and 95th percentile as around 1s, and this is happening on the replicas, it looks like we could raise the limit of MRs synced.
We should still have a limit though.
Finding the right balance between customer requirement and effects, perhaps the limit could be raised to 800 - 2 times higher, and would be batched in a max 2 Jira requests (due to the Jira APIs limit of 400 MRs per request).
We should work with Scalability team to look at this data and help us to assess what a limit should be.
The feature should be feature flagged.
We should keep the feature flag in the codebase for at least 1 milestone in order to assess the impact fully, as problems could happen only with customers with many projects and many MRs.