SDK should not require non-selected streams to be included in the catalog
Currently if you pass a catalog.json
to a tap created with Singer SDK, it will error if a stream is missing. I think if a stream is not included in the catalog
it should automatically be ignored for that sync.
Relevant line: https://gitlab.com/meltano/singer-sdk/-/blob/main/singer_sdk/streams/core.py#L382
Stacktrace
Traceback (most recent call last):
File "/Users/hassansyyid/env/testing-taps/bin/tap-procore", line 8, in <module>
sys.exit(cli())
File "/Users/hassansyyid/env/testing-taps/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/Users/hassansyyid/env/testing-taps/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/Users/hassansyyid/env/testing-taps/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/hassansyyid/env/testing-taps/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/Users/hassansyyid/env/testing-taps/lib/python3.7/site-packages/singer_sdk/tap_base.py", line 247, in cli
tap.sync_all()
File "/Users/hassansyyid/env/testing-taps/lib/python3.7/site-packages/singer_sdk/tap_base.py", line 178, in sync_all
for stream in self.streams.values():
File "/Users/hassansyyid/env/testing-taps/lib/python3.7/site-packages/singer_sdk/tap_base.py", line 64, in streams
stream.apply_catalog(self.input_catalog)
File "/Users/hassansyyid/env/testing-taps/lib/python3.7/site-packages/singer_sdk/streams/core.py", line 383, in apply_catalog
self.primary_keys = catalog_entry.key_properties
AttributeError: 'NoneType' object has no attribute 'key_properties'