Document unofficial Singer extension: `ACTIVATE_VERSION`
As explained by Jeff Huth in Slack:
Most taps are upsert only; either
FULL_TABLE
(all records) orINCREMENTAL
(only records on/after the bookemark). But there are a few taps out there that useACTIVATE_VERSION
. This is an undocumented feature and only works if the target is set up to deal with this... basically, it loads ALL records with a version, then (logically?) deletes earlier version(s) of the records with the same Primary Key. These allow you to upsert + delete (to logically delete the "delta" records).Imagine you load 100 records from a table today; then you delete 5 records and load 95 records tomorrow. This feature "deletes" the delta 5 records.
Most database targets allow for this. But some file-based targets and even singer-check-tap target fail when you try to run these taps.
tap-google-sheets
is a tap that uses this.
The ACTIVATE_VERSION
message is not currently documented in the the Singer spec (issue to change that: https://github.com/singer-io/getting-started/issues/71), but it is included in the singer-python
library (where it's marked as experimental): https://github.com/singer-io/singer-python/blob/master/singer/messages.py#L137
While this feature may be experimental from the Singer spec's perspective, it is implemented by at least one tap/target pair that's supported out of the box by Meltano, so it's effectively standardized and worth documenting:
- https://meltano.com/plugins/extractors/salesforce.html
- https://meltano.com/plugins/loaders/postgres.html
We are also planning to support this feature in taps and targets built using the Singer SDK: https://gitlab.com/meltano/singer-sdk/-/issues/18