Skip to content

Support variant `object` types in tap SCHEMA

As described in !120 (merged)

Per discussions in this thread:

  1. We do allow variant objects and we will merge MRs that allow this to "just work" in a tap.
  2. We do not recommend that tap developers use this practice until it has broader compatibility with targets.
  3. We do not have robust testing yet on the behavior of variant objects in regards to node selection.
  4. We expect #186 (closed) will introduce a DATATYPE_FAILSAFE capability (or similar) which will let targets advertise that they can accept these or similarly complex/ambiguous types.
  5. We expect #174 (closed) will allow inline schema detection, which will reduce the need for variant schemas in SDK-based taps.

@edgarrmondragon summarizes it well:

If the target doesn't support that, I would consider it a bug in the target (or missing a DATATYPE_FAILSAFE feature) since most databases are able to either manipulate JSON stored in text columns, or even have special column types. Even for targets like CSVs, the loader should be able to stringify the JSON with the right escaping.

Remaining todos:

  • Document the above best practices, recommendations, and considerations somewhere in sdk.meltano.com (low pri?)
  • Continue to expand test coverage, esp. node selection, for variant object types (low pri?)
Edited by AJ Steers