Use "setting.aliases" as alternative to "env_alias"
As raised today (2022-02-02
) in office hours, there is a benefit to env_aliases
in that they can be used to ease migration across forks of the same tap or target. I wanted to propose, along the same vein, a We have setting_alias
which would work as follows:settings.aliases
capability which in theory could work at a higher abstraction level:
- The alias would be referenceable in
meltano.yml
config:
blocks instead of the original name. - The alias could be used in env variable injection so that
TARGET_SNOWFLAKE_ACCOUNT
could be used in place ofTARGET_SNOWFLAKE_SNOWFLAKE_ACCOUNT
. - If both alias and the original name are set, this would be interpreted as a conflict. A hard failure would be raised: e.g.:
The setting 'SNOWFLAKE_ACCOUNT' was provided along with its alias name 'ACCOUNT', which is not permitted. Please specify 'SNOWFLAKE_ACCOUNT' or 'ACCOUNT' but not both.
Caveats
- As we're trying to move towards getting most/all config info from the hub (reducing dependency), this could be in conflict with that direction.
- Ideally, this would live in the hub definitions, rather than in discovery.yml, for the above reason.
Potential added benefit for user onboarding and switching between variants
There could be added benefit to aliases, in that certain cross-implementation representations could be implemented by Meltano (and other orchestrators) generically. Many taps have common config settings but are each subject to variations in naming.
Example config options which would make good candidates for generic and cross-implementation aliases:
account
username
password
port
auth_token
refresh_token
client_secret
host
s3_bucket
hard_delete
target_schema_name
target_db_name
If we allowed the Hub entry to declare any of these aliases/forms of settings in the hub metadata, then orchestrators could leverage this info, and also (back to one of the original benefits of env_aliases
), users could swap between variants without having to fully rewrite their config - at least not for those core elements that variants would have in common.
Benefits for dbt abstraction
Downstream tools like dbt
generally need a small number of variables related to where the data will land. Having common aliases for these functionally-significant parameters would allow Meltano to inject values and/or read values from those important settings. So, for instance, Meltano could know which setting names have the aliases target_schema_name
and target_db_name
, then we can use that to pass the values along to dbt for identifying the source of a transformation.