Formal handling of API rate limits
SDK error handling could be extended to support an APIRateLimitReached
exception type. The method for detecting rate limit errors would differ across implementations, but the shared benefit would be consistent error messaging to tap users.
Orchestrators such as Meltano might also be able to detect the specific error code or error message related to rate limits, and smartly retry after a specified interval (for instance, after an hour has passed). Similarly, we could add settings-based config for SDK-based taps, causing the tap to (1) flush the latest state messages, (2) sleep for a configurable interval, (3) retry after the sleep interval has passed, (4) as long as rate limits are reached, loop - until either a max runtime duration is reached or the process is terminated by the orchestrator.