Add request timeout for REST API taps
Summary
I recently had a pipeline that would hang indefinitely on random requests made to the api. Since there is no default timeout in requests
package, but they intend to add it in subsequent versions. I recommend we add a configurable timeout period with a default of 5(?) minutes on REST api requests.
Proposed benefits
This will provide tap developers feedback when their requests are taking too long and enable them to lengthen the timeout. Otherwise, it could hang indefinitely and make it difficult to debug like it did in my case.
Proposal details
This involves setting the timeout
parameter of the send
method of the prepared request in the _request
method of the RESTStream
class. I would also add the timeout exception error the backoff decorator. I would also recommend setting a class attribute in the cookiecutter template that calls this out as a configurable setting for new taps.
I will provide a PR for my recommendation.
Best reasons not to build
It would be an easy feature to miss when developing a new tap and could create frustration for healthy but long running api calls.