Provide access to last-seen record in get_next_page_token()

As @rabidaudio points out here:

Another thing I was thinking was it might be convenient for get_next_page_token to receive the latest row, as in this case and other taps I've written the next query starts from the timestamp of the last returned value, as opposed to the number of records or fixed amounts of paging.

And (after some hesitancy 😅) I replied:

... I think your intuition is good here. I have another tap implementation I was working on recently we have a since pagination method that takes the integer id of the last record... In both cases, we do need a value from the previous_record in order to orchestrate the pagination loop.

I've thought about this more, and I think the approach of passing the last-seen record's data is preferable to requiring the developer to stash the latest pagination data on each record's parsing, as @rabidaudio's implementation needed to do.

We could add latest_record or last_seen_record into get_next_page_token, with an implementation like:

    def get_next_page_token(self, response, previous_token, last_record: Optional[dict]):
        if not last_record:
            return None

        return { "since": last_record["id"] }

With this implementation, we can optionally also auto-detect if zero records are returned within a page that there is no subsequent page. Otherwise, the developer needs to provide logic for when to stop looping.

Also, we probably want last_record to be None if there have been no records emitted since the last pagination.


Updates (Jun 7, 2021):

  • I'm inclined to include this in an upcoming release. It will be a breaking change so it should be ideally grouped with other major and/or breaking changes.
  • My preference would be for last_seen_record to be None in the case of a fully empty page (meaning no records at all returned in the page). However, this makes naming the argument more difficult, since that behavior would not be obvious from the existing naming ideas we've logged.
Edited by AJ Steers