Nested properties in discovered catalog are interpreted as `inclusion: automatic` even if their parent property is `inclusion: available`
See https://gitlab.com/meltano/meltano/-/blob/master/src/meltano/core/plugin/singer/catalog.py#L284.
As a result, these nested properties will always show up as automatic
in meltano select --list --all
, cannot be deselected using meltano select --except
, and end up having metadata inclusion: automatic, selected: true
in the generated catalog file that's passed to the tap.
In reality, if a parent property is discovered as inclusion: available
, changing selected
or inclusion
on an individual subproperty usually won't make a difference since taps typically only check the selected
metadata for top-level properties, not nested ones.
To resolve this, then, we can have Meltano assume inclusion: automatic
on top-level properties that lack metadata, but not on nested properties, (assuming their parent property does have metadata). In meltano select --list --all
, nested properties without discovered(!) inclusion
metadata shouldn't be listed at all, since they can't actually be individually selected. If a nested property does have discovered inclusion
metadata, we can list it.
We could automatically have a top-level selected
rule cascade down to its properties, but let's not get into that for now.
As a temporary workaround, the following metadata
rule can be added to a plugin definition to flip these nested inclusion: automatic
metadatas to inclusion: available
:
extractors:
- name: tap-facebook
# ...
metadata:
'*':
'*.*':
inclusion: available