Use a JSON path to handle record extraction from API responses

Allows the SDK user to simply override the records_jsonpath and/or next_page_token_jsonpath class attribute of their RESTStream implementation to indicate how to extract records from the JSON response.

Explored options

jq

Pros

• Well maintained
• jq is a tool and syntax familiar to engineers working in a shell environment

Cons

• No wheel available for Windows

Example syntax

path	object	result
.	{"field": "value"}	[{"field": "value"}]
.[]	[{"field": "value"}]	[{"field": "value"}]
.data[]	{"data": [{"field": "value"}]}	[{"field": "value"}]

jsonpath-rw

Pros

• JSONPath is more or less of a standard
• No C-compilation or dedicated Windows wheels required

Cons

• Abandoned?

Example syntax

path	object	result
$	{"field": "value"}	[{"field": "value"}]
$[*]	[{"field": "value"}]	[{"field": "value"}]
$.data[*]	{"data": [{"field": "value"}]}	[{"field": "value"}]

Based on this Slack conversation: https://meltano.slack.com/archives/C01PKLU5D1R/p1619543806024500