Default `.env` conflicts with user provided loader connector configuration settings
What is the current bug behavior?
What is happening now?
When trying to save loader connector settings for target-postgres
they are persisted (in the database but the UI doesn't reflect this fact), but not leveraged in subsequent ELT pipeline schedules.
What is the expected correct behavior?
What should be happening?
Settings provided via the UI should be:
- reflected in the UI
- used in ELT
Steps to reproduce
How one can reproduce the issue?
- Install
target-postgres
- Set custom configuration and save
- Go back and click
target-postgres
's "Configure" button to display the configuration again - You will notice the custom settings are not reflected
@iroussos and I have come to realize that the various PG_...
settings in the .env
are being used instead. This is by design in terms of precedence from a backend implementation perspective, but the UI and ELT expectation of setting a custom configuration should work.
Possible fixes
- If I remove all the
PG_...
settings from the.env
the UI and ELT works as expected. I believe @mbergeron (and after chatting with @iroussos we seem to have a converging solution) suggests the fix as removing all or a portion of the following fromsrc/meltano/core/bundle/initialize.yml
:
.env: |
FLASK_ENV=development
SQLITE_DATABASE=meltano
PG_PASSWORD=warehouse
PG_USERNAME=warehouse
PG_ADDRESS=localhost
PG_SCHEMA=meltano
PG_PORT=5502
PG_DATABASE=warehouse
Here is a reference to the solution comment proposed in another issue !807 (comment 204147051)
Environment variables have utmost precedence. I think we probably have too many variables set in that file, but we still kinda need it for
dbt
IIRC, there is still some work to be done to properly inject variables from Meltano todbt
.
@iroussos would you be so kind to look into this, I'd be glad if we could remove the
.env
file altogether so that only developers/power users would use that to use.
Further regression test
Ensure we automatically catch similar issues in the future
-
Write additional adequate test cases and submit test results -
Test results should be reviewed by a person from the team