Selecting a standard meltano target for S3
Problem to solve
(Summarize the problem we are trying to solve in the form of we need to do [A], so that [B] can [C])
S3 is a commonly desired target, whether for a generic 'data lake' architecture or for newer generation warehouse platforms like Athena and Snowflake. Currently, Meltano does not have a designated OOTB S3 target.
Target audience
(For whom are we doing this? Include a persona)
Further details
(Include use cases, benefits, and/or goals)
Proposal
There are a few existing targets to choose from as a starting point:
- https://github.com/vitoravancini/target-s3
- https://github.com/faumel/target-s3-avro
- https://github.com/transferwise/pipelinewise-target-s3-csv
What does success look like, and how can we measure that?
(Define both the success metrics and acceptance criteria. Note that success metrics indicate the desired business outcomes, while acceptance criteria indicate when the solution is working correctly. If there is no way to measure success, link to an issue that will implement a way to measure this)
Regression test
(Ensure the feature doesn't cause any regressions)
-
Write adequate test cases and submit test results -
Test results should be reviewed by a person from the team
Links / references
Please note that this was taken from GitLab, to be changed accordingly