How can Meltano aid in post-load deduplication

Part of meltano#2504 (closed) is acknowledging that after loading there may be some duplication of records. Meltano should be able to help with deduplication whether through dbt or some other tooling.

--- old issue description. ---

Depending on the replication mode of a tap/target, it's not safe to assume that extracted data is unique in its raw form in the destination. This is solved in the modern stack by using dbt to create a "base" or "source" layer on top of the raw data to deduplicate and do simple cleaning of the data. We did this on the data team https://about.gitlab.com/handbook/business-ops/data-team/platform/dbt-guide/#source-models

It seems like it should be possible to have as part of the SDK a default base model generated for any data coming through the from the tap. Depending on the replication mode we'd need to pass (at least) the following:

  • primary key (could be surrogate)
  • updated_at timestamp

Not clear to me is if this would be a native part of the target or if it'd have to be a part of a separate package.

Edited by Taylor A Murphy