Draft: Allow skipping a partition by raising a specific exception
This PR is a proposal to address #282 (closed)
The change proposed here is fully opt-in:
- a new exception type
SkipPartitionAPIError
is added, which tap developers can raise to signify "this partition cannot be synced, but other partitions might, so continue with the next one". - unless that exception is raised, the tap's behaviour is unchanged,
- in particular, if
FatalAPIError
is raised, the tap will stop immediately.
Typical use case for this exception would be a server-side error on the API part which cannot be resolved with the tap, other than by retrying at a later date. We are seeing quite a lot of these on the github API, with no clear pattern or cause. In most cases, retrying within a few seconds is enough, but sometimes it is not. In that case, the entire tap run is currently cancelled, which seems a fairly high price to pay.
Obviously happy to have feedback on this and adjust to make it as useful as possible.
Closes #282 (closed)