Malformed schema_mapping config passed to target-snowflake
What is the current bug behavior?
What is happening now?
I'm trying to configure schema_mapping
for the transferwise snowflake target.
loaders:
- name: target-snowflake
variant: transferwise
pip_url: pipelinewise-target-snowflake==1.12.0
config:
account: my-account
dbname: MY_DATABASE
user: MY_USER
warehouse: MY_WAREHOUSE
file_format: MY_SCHEMA.CSV
role: MY_ROLE
schema_mapping:
public:
target_schema: RAW_PUBLIC
This results in this error:
target-snowflake | Traceback (most recent call last):
target-snowflake | File "/Users/dean/dev/hi-meltano/.meltano/loaders/target-snowflake/venv/bin/target-snowflake", line 8, in <module>
target-snowflake | sys.exit(main())
target-snowflake | File "/Users/dean/dev/hi-meltano/.meltano/loaders/target-snowflake/venv/lib/python3.8/site-packages/target_snowflake/__init__.py", line 438, in main
target-snowflake | table_cache, file_format_type = get_snowflake_statics(config)
target-snowflake | File "/Users/dean/dev/hi-meltano/.meltano/loaders/target-snowflake/venv/lib/python3.8/site-packages/target_snowflake/__init__.py", line 79, in get_snowflake_statics
target-snowflake | table_schemas=stream_utils.get_schema_names_from_config(config))
target-snowflake | File "/Users/dean/dev/hi-meltano/.meltano/loaders/target-snowflake/venv/lib/python3.8/site-packages/target_snowflake/stream_utils.py", line 43, in get_schema_names_from_config
target-snowflake | schema_names.append(target.get('target_schema'))
target-snowflake | AttributeError: 'str' object has no attribute 'get'
The original yaml config gets converted to json and passed to the snowflake target as a file: https://github.com/transferwise/pipelinewise-target-snowflake/blob/master/target_snowflake/init.py#L429
If I edit the plugin and log the contents of this file, I get:
{
"account": "my-account",
"dbname": "MY_DATABASE",
"user": "MY_USER",
"password": "hunter2",
"warehouse": "MY_WAREHOUSE",
"file_format": "MY_SCHEMA.CSV",
"default_target_schema": "TAP_POSTGRES",
"batch_size_rows": 100000,
"flush_all_streams": false,
"parallelism": 0,
"parallelism_max": 16,
"schema_mapping": {
"public.target_schema": "RAW_PUBLIC",
"public": {
"target_schema": "RAW_PUBLIC"
}
},
"disable_table_cache": false,
"add_metadata_columns": false,
"hard_delete": false,
"data_flattening_max_level": 0,
"primary_key_required": true,
"validate_records": false,
"no_compression": false,
"role": "MY_ROLE"
}
What is the expected correct behavior?
What should be happening?
schema_mapping
in the json should not include "public.target_schema": "RAW_PUBLIC",
, it should simply be
"schema_mapping": {
"public": {
"target_schema": "RAW_PUBLIC"
}
},
Steps to reproduce
How one can reproduce the issue?
python 3.8.9 meltano 1.73.0
Full meltano.yml
:
version: 1
send_anonymous_usage_stats: false
plugins:
extractors:
- name: tap-postgres
variant: transferwise
pip_url: pipelinewise-tap-postgres==1.7.1
config:
user: postgres
dbname: my-db
filter_schemas: public
default_replication_method: LOG_BASED
logical_poll_total_seconds: 5
select_filter:
- public-my_table
metadata:
public-my_table.*:
replication-method: FULL_TABLE
schema:
public-my_table:
export_file_exchange_id:
type:
- string
- 'null'
format: date-time
loaders:
- name: target-snowflake
variant: transferwise
pip_url: pipelinewise-target-snowflake==1.12.0
config:
account: my-account
dbname: MY_DATABASE
user: MY_USER
warehouse: MY_WAREHOUSE
file_format: MY_SCHEMA.CSV
role: MY_ROLE
schema_mapping:
public:
target_schema: RAW_PUBLIC
$ meltano install
$ meltano elt tap-postgres target-snowflake
Relevant logs and/or screenshots
Please use code blocks (```) to format console output
Possible fixes
If you can, link to the line of code that might be responsible for the problem or suggest a fix
Further regression test
Ensure we automatically catch similar issues in the future
-
Write additional adequate test cases and submit test results -
Test results should be reviewed by a person from the team