Append schema properties to catalog
Problem to solve
Some taps, such as tap-dynamodb, due to the nature of the data source, do not have the ability to inspect the schema of their streams. Instead, they simply report the schema of the stream as {"type":"object"}
unless a schema is specified in the catalog for the stream, at which point they will offer up that schema instead, effectively allowing the user to specify the schema if there is one.
However, meltano only overwrites schemas that are reported by discover, rather than appending or replacing them with what is defined by the meltano.yml
configuration. Since the tap can't inspect the schema, it's never reported by discover, so it doesn't make it into the catalog.
Proposal
Have meltano deep-merge meltano.yml
schema configs with those output by discover.
Regression test
(Ensure the feature doesn't cause any regressions)
-
Write adequate test cases and submit test results -
Test results should be reviewed by a person from the team
Links / references
See also this slack thread