Skip to content
GitLab
Next
    • Why GitLab
    • Pricing
    • Contact Sales
    • Explore
  • Why GitLab
  • Pricing
  • Contact Sales
  • Explore
  • Sign in
  • Get free trial
  • MeltanoMeltano
  • MeltanoMeltano
  • Issues
  • #2300

Allow extractor Singer messages to be transformed using Python before passing them to loader (Stream Maps)

We could support transformer plugins other than dbt, that would follow extraction rather than loading, and would transform an extractor's output stream of Singer messages (SCHEMA, RECORD, etc), before they're streamed into the loader.

The two types of transformers could be distinguished using an extra, e.g. type: {etl,elt} or {follows,acts_on,transforms}: {extractor,loader}

We could support both pip packages, and local executable files, using either the pip_url or executable plugin setting.

Edited Jun 11, 2021 by AJ Steers
Assignee
Assign to
Time tracking