[GPRD] Set TCP timeout parameters for rails' connection to the database

Production Change

Change Summary

Currently we don't set TCP timeout parameters for rails, which makes its DB connections susceptible to long blocking calls in case of untimely disconnects. We'll set those parameters in this change.

corrective action for https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/8528

Change Details

1. Services Impacted - ServicePostgres
2. Change Technician - @alejandro
3. Change Criticality - C1
4. Change Type - changeunscheduled
5. Change Reviewer - @ahmadsherif
6. Due Date - Date and time (in UTC) for the execution of the change
7. Time tracking - 91
8. Downtime Component - 0

Detailed steps for the change

Pre-Change Steps - steps to be completed before execution of the change

Estimated Time to Complete (mins) - 1

• Apply label changein-progress

Change Steps - steps to take to execute the change

Estimated Time to Complete (mins) - 45

• Merge and apply https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/247
• Run chef-client on rails-cny nodes: knife ssh -C 3 "roles:gprd-base-fe-web-cny OR roles:gprd-base-fe-api-cny" "sudo chef-client"
• Check that you're using the canary server at https://next.gitlab.com/ and that gitlab.com continues to work properly
• Run chef-client on all rails nodes: knife ssh -C 3 "roles:gprd-base-fe OR roles:gprd-base-be" "sudo chef-client"

Post-Change Steps - steps to take to verify the change

Estimated Time to Complete (mins) - 5

• Check that gitlab.com continues to work correctly

Rollback

Rollback steps - steps to be taken in the event of a need to rollback this change

Estimated Time to Complete (mins) - 30

• Rollback and apply https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/5264/
• Run chef-client on all rails nodes: knife ssh "roles:gprd-base-fe OR roles:gprd-base-be" "sudo chef-client"

Monitoring

Key metrics to observe

• Metric: 5xx responses
  • Location: https://dashboards.gitlab.net/d/RZmbBr7mk/gitlab-triage?orgId=1&refresh=30s
  • What changes to this metric should prompt a rollback: Degradation bellow SLOs
• Metric: Pgbouncer closed connection count
  • Location: https://log.gprd.gitlab.net/goto/585029deda5ec6fe1ab53cea7ce6ab78
  • What changes to this metric should prompt a rollback: Sustained increase in number of disconnects

Summary of infrastructure changes

• Does this change introduce new compute instances? No
• Does this change re-size any existing compute instances? No
• Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc? No

Summary of the above

Changes checklist

• This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities.
• This issue has the change technician as the assignee.
• Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed.
• Necessary approvals have been completed based on the Change Management Workflow.
• Change has been tested in staging and results noted in a comment on this issue.
• A dry-run has been conducted and results noted in a comment on this issue.
• SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall and this issue and await their acknowledgement.)
• There are currently no active incidents.