Handle unsupported metadata in singer taps
What is the current bug behavior?
Meltano currently (v1.39.1) ignores the metadata with "inclusion": "unsupported"
and try to include such attributes for an object causing the tap to fail.
What is the expected correct behavior?
Do not include metadata with "inclusion": "unsupported"
and conform with the Singer specification at https://github.com/singer-io/getting-started/blob/master/docs/DISCOVERY_MODE.md#metadata
Steps to reproduce
How one can reproduce the issue?
This issue can be seen with tap-zuora.
Add custom extractor tap-zuora and run meltano elt tap-zuora target-csv
.
Relevant logs and/or screenshots
CRITICAL There is a syntax error in one of the queries in the AQuA input (Query = select CreatedById, CreatedDate, EndDate, ErrorFileUrl, Id, ProcessedAccounts, ProcessedCharges, ResultFileUrl, RunNumber, StartDate, TotalAccounts, UpdatedById, UpdatedDate, Entity.Id from ChargeMetricsRun where UpdatedDate >= '1900-01-01T00:00:00 (+00:00)' order by UpdatedDate asc, Error = There is no field named EndDate)
For EndData the metadata produced by discovery is marked as unsupported:
{
"breadcrumb": [
"properties",
"EndDate"
],
"metadata": {
"inclusion": "unsupported"
}
}
Possible fixes
Ignore all unsupported fields when sending a query to tap
Further regression test
Ensure we automatically catch similar issues in the future
-
Write additional adequate test cases and submit test results -
Test results should be reviewed by a person from the team