Skip to content

Handling of API Rate limits

Fred O'Loughlin requested to merge fred-oloughlin-fl/sdk:fo-retry-on-rate-limiting into main

Addressing issue: 140 - The issue was partially resolved with the introduction of the backoff decorator in MR: 195, this builds on that work to add specific rate limit handling.

This change extends the 'RetriableAPIError' to accept the response as a parameter. This in turn can then be used as part of the backoff logic by overriding the pre_retry method.

It is not possible to pass the wait period to backoff wait parameters, as these wait times are evaluated at import. So to use the value from the response header, I'm using the on_backoff event that allows us to extend the backoff behaviour with a specific wait time using this value.

In order to catch unusual rate limiting codes, a stream could override RESTStream.rate_limit_response_codes, with other values.

The on_backoff event triggers as part of the backoff loop and immediately before the backoffs built in wait. So the backoff decorator is still handling the retry.

Any feedback or advice for improvement is appreciated.

Closes #140 (closed)

Closes #137 (closed)

Edited by Edgar R. Mondragón

Merge request reports