run_connection_test is pulling lots of data from my source

Diving into some tests today and noticed that the long running test was coming from the connection test function. MR incoming

To replicate

  1. Clone https://github.com/AutoIDM/tap-clickup
  2. Setup an API key for clickup (eek)
  3. Run poetry run pytest

You'll notice that each downstream task stream is ran to completion (pagination and all which takes a long time for large datasets)

stream._MAX_RECORDS_LIMIT = 1 if stream.child_streams else 0

Seems to be the issue

Extra details:

from singer_sdk.testing import get_standard_tap_tests

   def _test_stream_connections() -> None:
        # Initialize with basic config
        tap1: Tap = tap_class(config=config, parse_env_config=True)
        tap1.run_connection_test()
    @final
    def run_connection_test(self) -> bool:
        """Run connection test.

        Returns:
            True if the test succeeded.
        """
        for stream in self.streams.values():
            if stream.parent_stream_type:
                self.logger.debug(
                    f"Child stream '{type(stream).__name__}' should be called by "
                    f"parent stream '{stream.parent_stream_type.__name__}'. "
                    "Skipping direct invocation."
                )
                continue

            stream._MAX_RECORDS_LIMIT = 1 if stream.child_streams else 0
            try:
                stream.sync()
            except MaxRecordsLimitException:
                pass
        return True
Edited by Derek Visch