Enable :parallel_push_checks feature flag
Production Change
Change Summary
Enabling the :parallel_push_checks
feature flag to speed up the /api/v4/internal/allowed
endpoint. This will run push checks in separate threads (named push_check
for monitoring purposes) as the checks are heavily IO-bound. I expect to see a 30-40% speed improvement for this endpoint on slower requests.
Introduced in gitlab-org/gitlab!45668 (merged) Related gitlab-org/gitlab#222247 (closed)
Change Details
- Services Impacted - Rails, Postgres
- Change Technician - @robotmay_gitlab
- Change Criticality - C3
- Change Type - changescheduled
- Change Reviewer - DRI for the review of this change
- Due Date - Date and time (in UTC) for the execution of the change
- Time tracking - < 1 minute
- Downtime Component - None
Detailed steps for the change
- Enable
:parallel_push_checks
feature flag forwww-gitlab-com
project - Monitor
- Enable for
gitlab
project - Monitor
- Enable site-wide
Pre-Change Steps - steps to be completed before execution of the change
None
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - 1
-
/chatops run feature set parallel_push_checks true --project gitlab-com/www-gitlab-com
Post-Change Steps - steps to take to verify the change
Estimated Time to Complete (mins) - 60
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - 1
-
/chatops run feature set parallel_push_checks false --project gitlab-com/www-gitlab-com
Monitoring
Key metrics to observe
-
Metric:
gitlab_ruby_threads_running_threads
- Location: https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sum%20by%20(thread_name)%20(%20gitlab_ruby_threads_running_threads%7Buses_db_connection%3D%22yes%22%7D%20)&g0.tab=0
- What changes to this metric should prompt a rollback: Excessive numbers of threads
-
Metric:
Backend connections (web)
- Location: https://dashboards.gitlab.net/d/PwlB97Jmk/pgbouncer-overview?editPanel=17&orgId=1
- What changes to this metric should prompt a rollback: Excessive numbers of connections
-
Metric:
Waiting Client Connections
- Location: https://dashboards.gitlab.net/d/PwlB97Jmk/pgbouncer-overview?viewPanel=4&orgId=1
- What changes to this metric should prompt a rollback: Notable increase in waiting client connections
-
Metric:
Latency Apdex
- Location: https://dashboards.gitlab.net/d/web-main/web-overview?viewPanel=3&orgId=1
- What changes to this metric should prompt a rollback: Increase in overall response time from the Rails application (caused by blocking on database connections)
Summary of infrastructure changes
-
Does this change introduce new compute instances? -
Does this change re-size any existing compute instances? -
Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?
Summary of the above
Changes checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled). -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. -
Necessary approvals have been completed based on the Change Management Workflow. -
Change has been tested in staging and resultes noted in a comment on this issue. -
A dry-run has been conducted and results noted in a comment on this issue. -
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue.) -
There are currently no active incidents.
Edited by Robert May