Active Environment Name As Env Variable

Proposal

Would expose MELTANO_ENVIRONMENT to child processes, even if environment is set via another method (--environment at the CLI or default_environment in meltano.yml.)

Background

Based on this thread in slack https://meltano.slack.com/archives/C01TCRBBJD7/p1643919224929049

My original comment was:

I'm looking for a way to detect the active environment using environment variables and I know you can set a meltano environment using --environment=prod or through a MELTANO_ENVIRONMENT. Its easy to solve if I use the second method but if I define it using the --environment argument there doesnt seem to be any environment variable set. Is there a way to consistently do this?

We also now have default environments so the environment can be set in the meltano.yml too.

For any down stream process that wants to call meltano commands like I do in the dbt airflow dag generator https://gitlab.com/pnadolny13/files-airflow-dbt/-/blob/master/bundle/orchestrate/dags/generator_cache_builder.py#L36 it would be helpful to have the meltano environment exposed in the environment variables context. If I call meltano --environment=prod invoke airflow webserver then the dag generator should be able to access MELTANO_ENVIRONMENT.

It might be an anti pattern but it gives more flexibility to users so we can customize more.

added kindFeature valuestreamMeltano labels

marked this issue as related to #3105 (closed)

Marking as related to adapter-specific dbt as the file bundle files-dbt-snowflake could make use of MELTANO_ENVIRONMENT to support configuring the default profile from a selection of environment-specific profiles (as per this example).

@edgarrmondragon / @pandemicsyn / @tayloramurphy - Any reason not to do this? We expose other environment variables to child processes, and this one in particular seems important+necessary for many use cases.

(If anyone who knows the affected part of the codebase wants to take a stab on weight that would also be appreciated.)

@aaronsteers I think this is as easy as

1. Adding an environment entry to src/meltano/core/bundle/settings.yml

2. Overriding the value of that setting after the environment is activated in src/meltano/cli/cli.py:

   ProjectSettingsService.config_override["environment"] = active_environment

I'm inclined to make it a 4.

+1 @edgarrmondragon got it covered, should be pretty easy and afaik theres no reason not to do this.

Weight of 4 is applied. Thanks, @edgarrmondragon and @pandemicsyn .

Cc @tayloramurphy for prioritization when he's back in the office. (Tentatively putting into April milestone.)

changed the description

set weight to 4

changed milestone to %April 2022

mentioned in merge request files-dbt-snowflake!2 (merged)

mentioned in merge request !2593 (merged)

@kgpayne - I'm assigning to you for next week on the chance this is a blocker of files-dbt-snowflake!2 (comment 905245197). If not a blocker, we may be able to postpone/delay until a future iteration.

@aaronsteers I think this is worth shipping even if we launch files-dbt-snowflake without it as populating MELTANO_ENVIRONMENT unblocks at least one other important use case in addition to adapter-specific profile targets

assigned to @kgpayne

mentioned in issue #3298 (closed)

mentioned in merge request !2602 (merged)

added flowDoing label

added flowReview label and removed flowDoing label

mentioned in commit 6f1a3636

closed with merge request !2602 (merged)