Problems with environment variables taking precedence over declared ones

If a plugin configuration option is set to a specific environment variable, but that config option as a default environment variable it can be configured with, and both are defined, Meltano uses the default one rather than the explicitly declared one.

Example:

meltano init demo
meltano add extractor tap-gitlab
meltano config tap-gitlab set private_token '$MY_REAL_GITLAB_API_TOKEN'
export GITLAB_API_TOKEN=wrong-token
export MY_REAL_GITLAB_API_TOKEN=right-token
meltano config tap-gitlab
{
  "api_url": "https://gitlab.com",
  "private_token": "wrong-token",
  "groups": "",
  "projects": "",
  "ultimate_license": false,
  "fetch_merge_request_commits": false,
  "fetch_pipelines_extended": false
}

To me, this is unexpected behavior. I would expect GITLAB_API_TOKEN to be used if I hadn't specified anything for api_url, but since I explicitly told Meltano to use MY_REAL_GITLAB_API_TOKEN I'd expect that to take precedence.

Proposed Solution (AJ)

I think we should deprecate env: CUSTOM_ENV_NAME and env_aliases: [...] in favor of the more explicit <plugin_name>_<setting_name>. I've created #3120 (closed) to track this.

Edited by AJ Steers