<!-- Please review https://about.gitlab.com/handbook/engineering/infrastructure/change-management/ for the most recent information on our change plans and exection policies. --> # Production Change ### Change Summary Set the `SIDEKIQ_SEMI_RELIABLE_FETCH_TIMEOUT` to 5 on Sidekiq nodes. This will hopefully help alleviate some of the pressure caused by CPU saturation on redis-sidekiq in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4049 by reducing the overhead that setting up and tearing down connections causes ### Change Details 1. **Services Impacted** - ~"Service::Sidekiq" 1. **Change Technician** - @igorwwwwwwwwwwwwwwwwwwww @reprazent 1. **Change Criticality** - ~C3, <!-- Add this label to the issue itself, as well --> 1. **Change Type** - ~"change::unscheduled", ~"change::scheduled" 1. **Change Reviewer** - @reprazent 1. **Due Date** - Depends on https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57351 getting to production 1. **Time tracking** - {+ Time, in minutes, needed to execute all change steps, including rollback +} 1. **Downtime Component** - none ## Detailed steps for the change ### Pre-Change Steps - steps to be completed before execution of the change *Estimated Time to Complete (mins)* - {+Estimated Time to Complete in Minutes+} - [x] Make sure https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57351 is available on production - [ ] Get flamegraphs of the redis-sidekiq before the change ### Change Steps - steps to take to execute the change *Estimated Time to Complete (mins)* - {+Estimated Time to Complete in Minutes+} - [x] Merge https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/5238 that increases the timeout in staging. - [x] Merge https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/763 that increases the timeout for k8s workers in staging - [x] Report back here - [x] Merge https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/5240 that increases the timeout for catchall - [x] Merge https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/763 that increases the timeout for k8s workers in gprd ### Post-Change Steps - steps to take to verify the change *Estimated Time to Complete (mins)* - {+Estimated Time to Complete in Minutes+} - [x] Verify CPU saturation on https://dashboards.gitlab.net/d/redis-sidekiq-main/redis-sidekiq-overview?orgId=1&viewPanel=68 - [x] Get flamegraph of primary after the change ## Rollback ### Rollback steps - steps to be taken in the event of a need to rollback this change *Estimated Time to Complete (mins)* - {+Estimated Time to Complete in Minutes+} - [ ] Revert aforementioned MRs - [ ] {+Rollback Step 2+} - [ ] {+Rollback Step 3+} ## Monitoring ### Key metrics to observe <!-- * Describe which dashboards and which specific metrics we should be monitoring related to this change using the format below. --> - Metric: redis-sidekiq-cpu saturation - Location: https://dashboards.gitlab.net/d/redis-sidekiq-main/redis-sidekiq-overview?orgId=1&viewPanel=68 - Metric: Sidekiq apdex, error ratio, and RPS - Location: https://dashboards.gitlab.net/d/sidekiq-main/sidekiq-overview?orgId=1 ## Summary of infrastructure changes - [x] Does this change introduce new compute instances? No - [x] Does this change re-size any existing compute instances? No - [x] Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc? No <!-- * If you answer yes to any of the items in this checklist, summarize below. --> {+Summary of the above+} ## Changes checklist <!-- To find out who is on-call, in #production channel run: /chatops run oncall production. --> - [x] This issue has a criticality label (e.g. ~C1, ~C2, ~C3, ~C4) and a change-type label (e.g. ~"change::unscheduled", ~"change::scheduled") based on the [Change Management Criticalities](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#change-criticalities). - [ ] This issue has the change technician as the assignee. - [ ] Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. - [ ] Necessary approvals have been completed based on the [Change Management Workflow](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#change-request-workflows). - [ ] Change has been tested in staging and results noted in a comment on this issue. - [ ] A dry-run has been conducted and results noted in a comment on this issue. - [ ] SRE on-call has been informed prior to change being rolled out. (In #production channel, mention `@sre-oncall` and this issue and await their acknowledgement.) - [ ] There are currently no [active incidents](https://gitlab.com/gitlab-com/gl-infra/production/-/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=Incident%3A%3AActive).