Resolve "Introduce top-level "Environments" within `meltano.yml` config"

Merge Request Checklist

• Link to an issue
• Include the proposed fix or feature
• Include and update tests for the modified code
• Include a documentation change
• Add a CHANGELOG.md entry in the Unreleased section for any user-facing changes
• Mention or assign a maintainer

Spec collation

Based on #2869 (comment 686755671).

Environments will be implemented as a new SettingsStore.

meltano.yml (Updated 2021-10-15)

version: 1
plugins: ...
environments:
  - name: prod

    # The `destination` is the singular landing location ("target") for this environment
    # Other environments inherit this destination and can customize either with `vars` or by overriding completely
    primary_loader: target-snowflake

    # Environment-scoped plugin configuration
    config:
      plugins:
        loaders:
          - name: target-snowflake
            config:
              db_name: "${DB_NAME}"                              # References an environment-scoped env var
              load_schema: "${MELTANO_EXTRACTOR_NAMESPACE}_raw"  # Dynamic from the name of the currently-executing tap
              account: "${SNOWFLAKE_ACCOUNT}"
              user: "${SNOWFLAKE_USER}"
              role: prod_loader_role
              # ...

    # Environment-level config. Reusable across multiple plugins.
    env:
      DB_NAME: "Waffleshop_DB"
      SNOWFLAKE_ACCOUNT: waffleshop.snowflakecomputing.net
      SNOWFLAKE_USER: prod_loader

CLI

• In the following CLI commands, the --environment option can be used instead of the env var MELTANO_ENVIRONMENT like meltano <command> --environment=<environment> ....
• These commands will fail if prod is not known to Meltano.

invoke

# Overrides plugin configuration (`password`, `start_date`, etc.)
# and extras (`select`, `select_filter`, etc.)
MELTANO_ENVIRONMENT=prod meltano invoke my-tap

elt

# Overrides tap, target and dbt configurations
MELTANO_ENVIRONMENT=prod meltano elt my-tap target-snowflake

select ... --list (Updated 2021-10-18)

# Overrides tap extra configuration (`select`, `select_filter`, etc.) to determine selection
MELTANO_ENVIRONMENT=prod meltano select my-tap --list

config ... and config ... list (Updated 2021-10-15)

# Overrides dumped plugin configuration
MELTANO_ENVIRONMENT=prod meltano config my-tap

select ... (Updated 2021-10-19)

# Writes selection values to the right environment in `meltano.yml`
MELTANO_ENVIRONMENT=prod meltano select my-tap my_stream "*"

config ... set

# Writes config values to the right environment in `meltano.yml`
MELTANO_ENVIRONMENT=prod meltano config my-tap set my_config abc

Closes #2869 (closed)