Runner should respect Retry-After value in HTTP 429 responses from GitLab server
In some situations (namely migrations involving the ci_builds
table), it is important for the server to be able to tell the runner to back off for a bit while it is undergoing maintenance. A mechanism for that is returning an HTTP 429 Too Many Requests
response with a Retry-After
value of for example 5 minutes. Currently, our implementation of for example artifacts uploader will perform retries starting with 1 second and ending at 5 seconds:
1s
2s
4s
5s
5s
Program exited.
This is clearly not enough for a maintenance of the ci_builds
table, and has resulted in the need to revert migrations.