Allow plugins to inherit configuration from others
Use case
We have a ton of tables in a production database. As there are so many, to achieve hourly replication we split these tables into groups to run in parallel as separate 'pipelines'. Most of the configuration is the same for each extractor though, the only difference is in the select
stream identifier. To reduce the amount of duplication in the meltano.yml
file, allow plugins to inherit configuration from others, and then override with any explicitly set configuration values.
e.g.
plugins:
extractors:
- name: pipelinewise-tap-mysql-base
namespace: pipelinewise_tap_mysql
pip_url: git+https://github.com/tailsdotcom/pipelinewise-tap-mysql.git
executable: tap-mysql
load_schema: xxxxxxxx
capabilities:
- discover
- properties
- state
settings:
- name: user
env: PIPELINEWISE_TAP_MYSQL_USER
- name: password
env: PIPELINEWISE_TAP_MYSQL_PASSWORD
config:
host: xxxxxxxx
port: xxxxxxxx
dbname: xxxxxxxx
filter_dbs: xxxxxxxx
- name: pipelinewise-tap-mysql-1
inherit_from: pipelinewise-tap-mysql-base
select:
- a.*
- b.*
- c.*
...
metadata:
'*':
replication-method: FULL_TABLE