Stream Maps - Streams level __else__: null filter throws KeyError
Summary
Relates to: #301 (closed)
As part of finding a workaround for #301 (closed) I tried to use the __else__: null
stream map at the stream level to exclude all streams not explicitly defined in my stream map config. I get a KeyError when I include this.
Steps to reproduce
meltano, version 1.90.1
- name: target-athena-gitlab
inherit_from: target-athena
config:
stream_maps:
projects:
__alias__: "gitlab_projects"
users:
__alias__: "gitlab_users"
merge_requests:
__alias__: "gitlab_merge_requests"
issues:
__alias__: "gitlab_issues"
branches:
__alias__: "gitlab_branches"
commits:
__alias__: "gitlab_commits"
groups:
__alias__: "gitlab_groups"
project_labels:
__alias__: "gitlab_project_labels"
__else__: null
What is the current bug behavior?
I get a KeyError when I try to exclude all undefined streams like the docs describe.
What is the expected correct behavior?
All undefined streams are excluded with no errors.
Relevant logs and/or screenshots
2022-01-05T15:22:57.261721Z [error ] 'stream_maps.__else__'
Traceback (most recent call last):
File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/core/logging/output_logger.py", line 127, in redirect_logging
yield
File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/cli/elt.py", line 228, in _redirect_output
yield
File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/cli/elt.py", line 237, in _run_elt
elt_context = context_builder.context()
File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/core/elt_context.py", line 255, in context
env.update(loader.env)
File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/core/elt_context.py", line 28, in env
return {**self.plugin.info_env, **self.config_env()}
File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/core/elt_context.py", line 24, in config_env
return self.settings_service.as_env(session=self.session, **kwargs)
File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/core/settings_service.py", line 180, in as_env
full_config = self.config_with_metadata(*args, **kwargs)
File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/core/settings_service.py", line 147, in config_with_metadata
value, metadata = self.get_with_metadata(
File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/core/settings_service.py", line 233, in get_with_metadata
value, get_metadata = manager.get(name, setting_def=setting_def)
File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/core/settings_store.py", line 646, in get
value, metadata = manager.get(name, setting_def=setting_def, **kwargs)
File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/core/settings_store.py", line 501, in get
value, metadata = self.get_with_metadata(setting_def.name)
File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/core/settings_store.py", line 528, in get_with_metadata
metadata = self.config_with_metadata[name]
KeyError: 'stream_maps.__else__'
'stream_maps.__else__'
Possible fixes
Edited by Pat Nadolny