Skip to content

Requesting SRE advice on running Jira Connect sync script for 400+ customer projects

Details

  • Point of contact for this request: @anton @cms

Background

  • One of our SaaS customers is affected by a limitation in the GitLab for Jira Cloud app and would like to bypass the limitations for existing projects and sync all project data to Jira.
  • We currently have an open issue to address this limitation but it is in the backlog gitlab-org/gitlab#419736
  • To help this customer in the meantime, I wrote a workaround script as a courtesy https://gitlab.com/gitlab-com/support/toolbox/jira-connect-sync with the understanding we were going to execute the script on one project (and maybe a couple of others).
  • I've been working closely with @.luke from the Import and Integrate group and he has been reviewing the iterations on the script.
  • The customer has since pushed back and asked if we can run the script on 400+ projects.
  • The script was written with one project in mind and is quite resource intensive/takes a long time to run due to multiple N+1 issues, and that we aren't fetching commits from Gitaly in the most efficient way.

Customer

SRE Support Needed

We would like some advice from an SRE about the parameters of running this script for 400+ projects, whether it is a feasible idea or whether it's possible to stage it in a way that would work.

Please note I am on PTO now until April 2, but @cms will be the Support DRI until I return.

@.luke has been providing support to me when I wrote the script, so I'm hoping he can help with the conversation here too.

Estimated script duration

NOTE: This is indicative only.

In https://gitlab.com/gitlab-com/support/internal-requests/-/issues/25947#note_2409364174, we pulled their project metrics and then asked Anthopic the following:

<PASTED JSON DATA https://gitlab.com/gitlab-com/support/internal-requests/-/issues/25947#note_2409364174>

<PASTED SCRIPT https://gitlab.com/gitlab-com/support/toolbox/jira-connect-sync/-/raw/main/jira_connect_sync.rb>

<PASTED BENCHMARKS FROM DEV https://gitlab.com/gitlab-com/support/toolbox/jira-connect-sync/-/blob/main/README.md#performance-benchmarks>

<PASTED ACTUAL SCRIPT EXECUTION FROM PRODUCTION https://gitlab.com/gitlab-com/support/internal-requests/-/issues/25856#note_2389620684>

Read the json output (branches wasn't included unfortunately) and give a rough estimate on how long this will take based on the benchmarks. The dev benchmarks are much faster than production, and we will be executing this in production. Also keep in mind that the commit_count is indicative and that we process 15 commits from each branch.

Give me a small sumamry.

Based on the provided data and benchmarks, I can estimate the time required to run the Jira Connect synchronization script on this GitLab project collection.

## Summary of Repository Data
- 310 repositories in total
- Total commits across repositories: ~260,000
- Total merge requests across repositories: ~59,000
- Total CI pipelines across repositories: ~95,000
- Largest repositories by size: 2.6GB, 2.0GB, 1.7GB
- Total feature flags and deployments appear minimal

## Time Estimate for Production Environment

From the production execution log, we can see:
- Branches sync (1,733 branches): Completed in about 5-10 minutes
- Commits sync: Took approximately 3.5 hours for the same repository

**Estimated total sync time for all 310 repositories:**
- Merge requests: ~8-10 hours
- Pipelines: ~12-15 hours
- Branches: ~12-15 hours
- Commits: ~70-90 hours (This is the most time-consuming operation)
- Deployments & feature flags: ~1-2 hours

**Total estimated time: 100-130 hours** (approximately 4-5.5 days if run sequentially)

Other notes