add --setup method to taps and targets

Many taps and targets require setup steps to be performed before pipelines will run successfully. This typically includes:

  • Configuring databases to decode binlogs (e.g. MySQL and Postgres)
  • Configuring intermediate storage for targets (e.g. S3 for Snowflake and Redshift)
  • Checking connectivity (esp. VPC/Security Groups and non-AWS equivalents)
  • Checking user permissions

Today these steps can generally be found in documentation and readme's, and are performed either manually or in other automation frameworks (terraform, permifrost etc.). It can be difficult managing 10's of databases (as a monolith becomes microservices) across multiple environments (staging and production), as well as being a barrier to easy on-boarding for new users. I can also see 'structured setup' being useful for auto-generating documentation in SingerHub.

It would be great to have two facilities via a --setup method:

  • Run setup steps interactively (asking for input and credentials as needed)
  • Check the setup status of a tap or target (e.g. is bin-log replication enabled? have snowflake stages been created?)