Entity selection does not work when stream names have periods in them
What is the current bug behavior?
I’ve installed a custom plugin for redshift as an extractor (pip install tap-redshift
). I had to do some debugging of their code but I got it working all except for the entity selection. When I do a meltano select tap-redshift --list --all
, I get a list of all the tables in the schema
set in my config. However, when I try to add model selection to the meltano.yml
file using the expected syntax (i.e., database.schema.table.*
), I can’t get any of the entities in that list highlighted in green (‘selected’) no matter what I do. It almost feels like the select cli command doesn’t play nice with custom plugins.
What is the expected correct behavior?
If I enter
select:
- database.schema.table.*
I would expect all columns in the listed table to be selected in green ('selected') after running a meltano select --list --all
commmand.
Steps to reproduce
Here's my meltano.yml
version: 1
plugins:
extractors:
- name: tap-redshift
namespace: tap_redshift
pip_url: tap-redshift
executable: tap-redshift
capabilities:
- config
- discover
- catalog
- properties
config:
host: ############
port: 5439
dbname: matillion
schema: referencedata
start_date: 1900-01-01T01:01:01Z
user: ########
password: #########
select:
- 'matillion.referencedata.t_ref_score_range.*'
metadata:
'*':
replication-method: FULL_TABLE
Relevant logs and/or screenshots
% meltano select tap-redshift --list --all
Legend:
selected
excluded
automatic
Enabled patterns:
matillion.referencedata.t_ref_score_range.*
Selected attributes:
[excluded ] matillion.referencedata.t_ref_code_set.code_set_code
[excluded ] matillion.referencedata.t_ref_code_set.code_set_id
[excluded ] matillion.referencedata.t_ref_code_set.code_set_name
[excluded ] matillion.referencedata.t_ref_code_set.date_modified
[excluded ] matillion.referencedata.t_ref_code_set.modified_by
[excluded ] matillion.referencedata.t_ref_code_set.owner_name
[excluded ] matillion.referencedata.t_ref_code_set.steward_name
[excluded ] matillion.referencedata.t_ref_score_range.date_modified
[excluded ] matillion.referencedata.t_ref_score_range.lower_bound
[excluded ] matillion.referencedata.t_ref_score_range.metric_name
[excluded ] matillion.referencedata.t_ref_score_range.score_range_id
[excluded ] matillion.referencedata.t_ref_score_range.score_value
[excluded ] matillion.referencedata.t_ref_score_range.upper_bound
On a separate note, the workaround I found was to use quotes and ignore the first couple of dots in the entity name. Specifically:
select:
- '*t_ref_score_range.*'
gets me exactly what I want.
Possible fixes
If you can, link to the line of code that might be responsible for the problem or suggest a fix Not sure