Standardize the database connection
Context
Databases should be treated as a Meltano concept, typically defined as an entry in the meltano.yml
.
As Meltano uses SQLAlchemy behind the hood, it is natural to use the dialect[+driver]://user:password@host/dbname[?key=value..]
scheme as a naming convention across all database connections.
databases:
- name: warehouse
namespace: sqlite
settings:
- name: dialect
value: sqlite
- name: user
value: warehouse
- name: password
value: warehouse
- name: host
value: 'localhost:5432'
- name: dbname
value: warehouse
Iteration path
- Refactor database connection as a Meltano plugin
- Refactor the model connection to use the Database definition
- Generate loader configuration using
Databases
, via thekind
. For instance,target-sqlite
could have these settings:
name: target-sqlite
namespace: sqlite
settings:
- name: database
kind: sqlite
We'll have to think carefully where we want the mapping (my guts tells me namespace => kind
) to happen, but this would alleviate much of the UX pain for the loaders.
- Expose the meltano system database?