Retry HTTP request is 5xx error is encountered

Within 2022-11-02: Intermittent Internal API unreachable (gitlab-com/gl-infra/production#7979 - closed) investigation it's been noticed that gitlab shell reports a lot of Internal API unreachable errors:

There is a suggestion to implement retry logic that retries a request when 5xx error is encountered.

Let's investigate it within this issue.

Reports

(Within 7 days)

screenshot