Skip to content

Draft: Allow skipping a partition by raising a specific exception

Laurent Savaëte requested to merge LaurentS/sdk:skip-partition-on-error into main

This PR is a proposal to address #282 (closed)

The change proposed here is fully opt-in:

  • a new exception type SkipPartitionAPIError is added, which tap developers can raise to signify "this partition cannot be synced, but other partitions might, so continue with the next one".
  • unless that exception is raised, the tap's behaviour is unchanged,
  • in particular, if FatalAPIError is raised, the tap will stop immediately.

Typical use case for this exception would be a server-side error on the API part which cannot be resolved with the tap, other than by retrying at a later date. We are seeing quite a lot of these on the github API, with no clear pattern or cause. In most cases, retrying within a few seconds is enough, but sometimes it is not. In that case, the entire tap run is currently cancelled, which seems a fairly high price to pay.

Obviously happy to have feedback on this and adjust to make it as useful as possible.

Closes #282 (closed)

Edited by Edgar R. Mondragón

Merge request reports