Unbounded Transaction Duration Inside Rails
Summary
In production#18677 (closed) we saw a single query executed from the rails console that took over 30 minutes to execute (before it was terminated):
This resulted into a service wide degradation.
Related Incident(s)
Originating issue(s):production#18677 (closed)
Current Status
gitlab-com/gl-infra/data-access/dbo&10
Desired Outcome/Acceptance Criteria
The application has a 15-second timeout on queries. It's understandable that we might need this to be higher in the Rails console since we usually use that for debugging, ad-hoc queries. However, we should still prevent very long transactions from happening in the Rails console or anywhere else (production#18701 (closed)).
We should have a hard limit on database transaction everywhere (application, console) where we terminate any transaction that is longer then 8 minutes (for example) and log this somewhere to debug/triage at a later stage async.
Corrective Action Issue Checklist
- Link the incident(s) this corrective action arose from
- Give context for what problem this corrective action is trying to prevent re-occurring
- Assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4')
- Assign a priority (this will default to 'Reliability::P4' but should match the severity of the related incident)
- Assign a service label
- Assign a team label