How can Meltano aid in post-load deduplication
Part of meltano#2504 (closed) is acknowledging that after loading there may be some duplication of records. Meltano should be able to help with deduplication whether through dbt or some other tooling.
--- old issue description. ---
Depending on the replication mode of a tap/target, it's not safe to assume that extracted data is unique in its raw form in the destination. This is solved in the modern stack by using dbt to create a "base" or "source" layer on top of the raw data to deduplicate and do simple cleaning of the data. We did this on the data team https://about.gitlab.com/handbook/business-ops/data-team/platform/dbt-guide/#source-models
It seems like it should be possible to have as part of the SDK a default base model generated for any data coming through the from the tap. Depending on the replication mode we'd need to pass (at least) the following:
- primary key (could be surrogate)
- updated_at timestamp
Not clear to me is if this would be a native part of the target or if it'd have to be a part of a separate package.