Skip to content

Stream Maps - Streams level __else__: null filter throws KeyError

Summary

Relates to: #301 (closed)

As part of finding a workaround for #301 (closed) I tried to use the __else__: null stream map at the stream level to exclude all streams not explicitly defined in my stream map config. I get a KeyError when I include this.

Steps to reproduce

meltano, version 1.90.1

  - name: target-athena-gitlab
    inherit_from: target-athena
    config:
      stream_maps:
        projects:
          __alias__: "gitlab_projects"
        users:
          __alias__: "gitlab_users"
        merge_requests:
          __alias__: "gitlab_merge_requests"
        issues:
          __alias__: "gitlab_issues"
        branches:
          __alias__: "gitlab_branches"
        commits:
          __alias__: "gitlab_commits"
        groups:
          __alias__: "gitlab_groups"
        project_labels:
          __alias__: "gitlab_project_labels"
        __else__: null

What is the current bug behavior?

I get a KeyError when I try to exclude all undefined streams like the docs describe.

What is the expected correct behavior?

All undefined streams are excluded with no errors.

Relevant logs and/or screenshots

2022-01-05T15:22:57.261721Z [error    ] 'stream_maps.__else__'         
Traceback (most recent call last):
  File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/core/logging/output_logger.py", line 127, in redirect_logging
    yield
  File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/cli/elt.py", line 228, in _redirect_output
    yield
  File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/cli/elt.py", line 237, in _run_elt
    elt_context = context_builder.context()
  File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/core/elt_context.py", line 255, in context
    env.update(loader.env)
  File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/core/elt_context.py", line 28, in env
    return {**self.plugin.info_env, **self.config_env()}
  File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/core/elt_context.py", line 24, in config_env
    return self.settings_service.as_env(session=self.session, **kwargs)
  File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/core/settings_service.py", line 180, in as_env
    full_config = self.config_with_metadata(*args, **kwargs)
  File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/core/settings_service.py", line 147, in config_with_metadata
    value, metadata = self.get_with_metadata(
  File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/core/settings_service.py", line 233, in get_with_metadata
    value, get_metadata = manager.get(name, setting_def=setting_def)
  File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/core/settings_store.py", line 646, in get
    value, metadata = manager.get(name, setting_def=setting_def, **kwargs)
  File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/core/settings_store.py", line 501, in get
    value, metadata = self.get_with_metadata(setting_def.name)
  File "/Users/pnadolny/.virtualenvs/squred/lib/python3.8/site-packages/meltano/core/settings_store.py", line 528, in get_with_metadata
    metadata = self.config_with_metadata[name]
KeyError: 'stream_maps.__else__'
'stream_maps.__else__'

Possible fixes

Edited by Pat Nadolny