Support Secrets Backends Natively

Problem to solve

Meltano should support multiple secrets backends so that secret config values can be managed in the same way that general config is.

Meltano provides a convenient, centralised configuration management and injection layer for managing all the tools in the data science workflow. However this doesn't currently include secret config values such as passwords, tokens and keys. This sensitive data must currently be handled by the user outside of meltano, with secrets being injected at runtime using environment variables. This is both an inconvenience for regular users and a barrier to entry for people new to meltano.

Target audience

Anyone deploying meltano in a production environment.

Proposal

Something like:

# Configure secret values just as you configure normal values
meltano config <plugin> set --secret <key> <value>

Implementation thoughts:

• The secrets backend should be pluggable, to support all the different places secrets can be stored today (AWS Parameter Store, Hashicorp Vault, git-crypt etc.).
• Meltano should supply a default secrets backend, with either file-based encryption (git-crypt) or its database (with password hashing, ala Airflow).

What does success look like, and how can we measure that?

Users can manage all config through meltano. Security best practices are built-in rather than left as an exercise to the user.