Tap Validate Option
Summary
Validate tap data schema, particularly useful during development as right now you have to add a target that does this validation ie meltano elt tap-abc target-jsonl jsonl will validate for you. Maybe it should stay like this?
Proposed benefits
- One less step while doing development, the issue I'm seeing is running production data through a tap I'm writing leads to a bunch of Data in a format that wasn't expected (String is sometimes an Int, or something it's an array of Strings, etc etc).
- Theoretically you could suggest a new schema for the stream that's causing issues.
Proposal details
A --validate option
Standard validate_json config
Best reasons not to build
- You could offload this to a target
- People may forget to turn this off in production which would have performance implications
Use Cases
Edited by Derek Visch