select does not seem to limit columns pulled from tap
What is the current bug behavior?
select command, users can indicate a specific column for the table they are selecting. This is captured correctly in
meltano.yml. However, when running the elt process, all of the columns are selected rather than just the ones specified.
What is the expected correct behavior?
Only the indicated columns should be used when pulling data via the tap.
Steps to reproduce
How one can reproduce the issue?
select indicate a table and a particular column rather than '*'.
elt and view the query executed in the output.
Refer to the table created and note it contains all columns from the original table.
Relevant logs and/or screenshots
meltano select tap-postgres "flyway_schema_history" "installed_rank" and then
meltano elt tap-postgres target-postgres and saw the following in the output:
INFO Loading 44 rows into 'flyway_schema_history' INFO COPY flyway_schema_history_temp ("checksum", "description", "execution_time", "installed_by", "installed_on", "installed_rank", "script", "success", "type", "version") FROM STDIN WITH (FORMAT CSV, ESCAPE '\') INFO UPDATE 0 INFO INSERT 0 44
I also confirmed that all the columns were pulled from the source.
If you can, link to the line of code that might be responsible for the problem or suggest a fix
Further regression test
Ensure we automatically catch similar issues in the future
- Write additional adequate test cases and submit test results
- Test results should be reviewed by a person from the team