CommitDiff fails to respond for certain shas
Problem
CommitDiff
is called in an API route gitlab#331326 (comment 578914281). The RPC consistently fails to return a response for certain sha
s within the timeout deadline.
We can see a list of most commonly failing sha
s in the path
segments of this table: https://log.gprd.gitlab.net/goto/20f36b58673cb8e0fac06044257038e2.
Debugging on production gitlab#331326 (comment 641471224), it looked like even when the limits were turned right down and the timeout was increased four-fold, that the RPC would still fail to return a response gitlab#331326 (comment 641521165).
We can see that generally the RPC will either return data very quickly or fail to return within the timeout limit gitlab#331326 (comment 644597324), there's not any real graduation.
This problem affects other parts of the app besides this API endpoint gitlab#331326 (comment 640446821) however, the endpoint is called very frequently, so this issue disproportionately affects it.
Debugging
We can refer to the top offending sha
s in this table https://log.gprd.gitlab.net/goto/20f36b58673cb8e0fac06044257038e2 as a starting point for debugging.