run_connection_test is pulling lots of data from my source
Diving into some tests today and noticed that the long running test was coming from the connection test function. MR incoming
To replicate
- Clone https://github.com/AutoIDM/tap-clickup
- Setup an API key for clickup (eek)
- Run
poetry run pytest
You'll notice that each downstream task stream is ran to completion (pagination and all which takes a long time for large datasets)
stream._MAX_RECORDS_LIMIT = 1 if stream.child_streams else 0
Seems to be the issue
Extra details:
from singer_sdk.testing import get_standard_tap_tests
def _test_stream_connections() -> None:
# Initialize with basic config
tap1: Tap = tap_class(config=config, parse_env_config=True)
tap1.run_connection_test()
@final
def run_connection_test(self) -> bool:
"""Run connection test.
Returns:
True if the test succeeded.
"""
for stream in self.streams.values():
if stream.parent_stream_type:
self.logger.debug(
f"Child stream '{type(stream).__name__}' should be called by "
f"parent stream '{stream.parent_stream_type.__name__}'. "
"Skipping direct invocation."
)
continue
stream._MAX_RECORDS_LIMIT = 1 if stream.child_streams else 0
try:
stream.sync()
except MaxRecordsLimitException:
pass
return True
Edited by Derek Visch