Change Admin Import/Export Rate Limits
Production Change
Change Summary
We previously lowered the limit in #3050 (closed) due to high project export latency sidekiq queue as outlined in gitlab-org/gitlab#284498 (comment 450056343) with settings documented in https://docs.gitlab.com/ee/user/gitlab_com/#importexport
I have confirmation from dev in gitlab-org/gitlab#288833 (comment 484343065) that since we moved project template exports to a separate sidekiq queue in gitlab-org/gitlab#285054 (closed) that we should no longer need these lower settings.
As the lower limits were meant to be a temporary change until that work was completed, I'm proposing we change the settings back to the default group and default project limits.
Change Details
- Services Impacted - GitLab.com setting
- Change Technician - SRE on call (@craig)
- Change Criticality - C3
- Change Type - changeunscheduled
- Change Reviewer - Import group @georgekoltsov
- Due Date - ASAP
- Time tracking - 25
- Downtime Component - N/A
Detailed steps for the change
Pre-Change Steps - steps to be completed before execution of the change
Estimated Time to Complete (mins) - 15
-
Have SRE double check proposed limits
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - 5
-
Go to GitLab.com Admin Settings > Network > Import/Export Rate Limits - Max Project Export requests per minute per user: 6
- Max Group Export requests per minute per user: 6
Post-Change Steps - steps to take to verify the change
Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes
- [-] N/A
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - 5
-
Change settings back to previous settings - Max Project Export requests per minute per user: 1
- Max Group Export requests per minute per user: 1
Monitoring
Key metrics to observe
- Metric: scheduling latency
- Location: https://log.gprd.gitlab.net/goto/2db559d69770c6369fa007f3204e4c3e
- Similar to the previous issue #3050 (comment 450402375) , we'll want to keep an eye on whether there is an increase in long (1+ hour) scheduling delay
Summary of infrastructure changes
- [-] Does this change introduce new compute instances?
- [-] Does this change re-size any existing compute instances?
- [-] Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?
Changes checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities. -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. - [-] Necessary approvals have been completed based on the Change Management Workflow.
- [-] Change has been tested in staging and results noted in a comment on this issue.
- [-] A dry-run has been conducted and results noted in a comment on this issue.
-
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue.) -
There are currently no active incidents.