More granular set of capabilities around catalog input
SDK-based taps support the following catalog overrides and metadata:
- Select streams
- Set other stream and property metadata
replication-method,replication-key,is-replication-key, etc. - Select stream properties
- Override stream schema
However, other taps out there don't support all of the above and usually they only support the first two at best.
Ideally, all taps should support rich metadata and schema overrides and the SDK will help us get there, but in the meantime I propose a more granular set of documented capabilities around the catalog:
catalog:select-streamscatalog:select-propertiescatalog:metadata-streamscatalog:metadata-propertiescatalog:schema-override
Another question is whether we should make Meltano itself aware of these capabilities and warn users when they're using schema with a tap that doesn't support schema overrides, for example.
Edited by Edgar R. Mondragón