Make maximum numbers of API retries configurable
We have a long-running Terraform pipeline that sometimes fails due to 429 (Too Many Requests) responses from gitlab.com, especially when another pipeline that is also creating a lot of GitLab API requests is running from the same set of public IPs.
Our other curl-based pipeline is able to handle 429 responses fine and respect the Retry-After
Header set on the response. It takes longer to complete due to the rate-limiting, but does not fail.
The Terraform provider should do this as well as it's implemented by the GitLab API client:
https://github.com/xanzy/go-gitlab/blob/83e28c21ec3677ba531694456ed8c18400a9e796/gitlab.go#L440-L496
However our pipeline using this provider still fails regularly. The API client also sets a default maximum number of retries of 5, which we suspect is being hit faster due to Terraform's parallelism. It's also possible that the provider, after having waited until the time specified in the Retry-After
Header, gets another 429 response after the rate-limit for our public IPs has been extended/reinstated due to traffic from another pipeline.
This MR makes the maximum number of retries configurable via a http_max_retries
option on the provider. It keeps the default of 5 from the API client.