spec loader FAILS when there are more than 10000 tables in one database/schema
Describe the bug
While running permifrost when load the spec file and check that all entities exist in snowflake, in case we have large schema this fails. As SHOW TABLES TABLES IN DATABASE command over all tables is a really wide search and we have a lot of tables in stage db which we do not want to get rid of soon, while validating the table names mentioned in the spec file we have been facing this issue -
Checking that all entities in the spec file are defined in Snowflake snowflake.connector.errors.ProgrammingError: 090153 (22000): The result set size exceeded the max number of rows(10000) supported for SHOW statements. Use LIMIT option to limit result set to a smaller number.
Steps To Reproduce
Have one Database or Schema with more than 10k tables and try to run permifrost dry run command
Expected behavior
We should be able to validate if the table or database or schema exists without having to run SHOW command on all of it
Screenshots and log output
The output of permifrost --version:
permifrost, version 0.14.0The operating system you're using:
Mac OS
The output of python --version:
Python 2.7.16
Additional context
The solution proposed in the branch 'hkaur5.bug.update-check-table-entities' improves the library, but we still have issues with big schemas.