Change Admin Import/Export Rate Limits
Production Change
Change Summary
We are seeing a high project export latency sidekiq queue.
As per gitlab-org/gitlab#284498 (comment 450056343)
we should consider lowering the rate limit temporarily to see if that helps the situation.
Change Details
- Services Impacted - GitLab.com setting
- Change Technician - SRE on call
- Change Criticality - C3
- Change Type - changeunscheduled
- Change Reviewer - Import group @georgekoltsov
- Due Date - ASAP
- Time tracking - 25
- Downtime Component - N/A
Detailed steps for the change
Pre-Change Steps - steps to be completed before execution of the change
Estimated Time to Complete (mins) - 15
-
Have SRE double check proposed limits
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - 5
-
Go to GitLab.com Admin Settings > Network > Import/Export Rate Limits - Max Project Export requests per minute per user: 1
- Max Group Export requests per minute per user: 1
Post-Change Steps - steps to take to verify the change
Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes
- [-] N/A
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - 5
-
Change settings back to default
Monitoring
Key metrics to observe
- Metric: scheduling latency
- Location: https://log.gprd.gitlab.net/goto/2db559d69770c6369fa007f3204e4c3e
- What changes to this metric should prompt a rollback: Nothing? If it doesn't help, we'll want to lower the settings further.
Summary of infrastructure changes
- [-] Does this change introduce new compute instances?
- [-] Does this change re-size any existing compute instances?
- [-] Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?
Changes checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities. -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. - [-] Necessary approvals have been completed based on the Change Management Workflow.
- [-] Change has been tested in staging and results noted in a comment on this issue.
- [-] A dry-run has been conducted and results noted in a comment on this issue.
-
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue.) -
There are currently no active incidents.
Edited by Matt Smiley