Skip to content

Resolve "Allow multiple `--config=` inputs, parse from env vars only if `--config=ENV` is set"

AJ Steers requested to merge 53-multiple-config-inputs into development

Closes #53 (closed)

TL;DR:

This MR adds the ability for CLI arg --config to accept multiple inputs, including a new --config=ENV setting which automatically captures settings from environment variables. Breaking change: parse_env_config no longer defaults to True, and instead must be set explicitly in the CLI or in the class constructor. See "Breaking Changes" below for more info.

Details:

  • If one of the supplied inputs is --config ENV (or --config=ENV according to the user's preference), the environment variable parsing rules will be applied to ingest config values from environment variables.
  • One or more files can also be sent to --config. If multiple files are sent, they will be processed in sequential order.
  • If one or more files conflict for a given setting, the latter provided files will override earlier provided files.
    • This behavior allows to you easily "inject" environment overrides by adding --config=path/to/overrides.json at the end of the CLI command text.
  • If --config=ENV is set and one or more files conflict with an environment variable setting, the environment variable setting will always have precedence, regardless of ordering.
  • One new benefit of this approach is that credentials and other secrets can be stored completely separately from general settings: either by having two distinct config.json files or by using environment variables for secure settings and config.json files for the rest.

IMPORTANT: Breaking Change:

  • Prior to this MR, taps automatically collected environment variables regardless of inputs and the Tap constructor sent parse_env_config=True by default. Now, the setting must be called explicitly - either by setting --config=ENV at the CLI or by specifying parse_env_config=True in the Tap class constructor.
Edited by AJ Steers

Merge request reports