Hub metadata for plugins should be a superset of necessary fields in discovery.yml

As a path to unblock Add lock artifacts during `meltano add` for sta... (meltano#3031 - closed) and also to replace discovery.yml in the Meltano project with the metadata here in the Hub, we'd like to ensure that Hub metadata is a superset of the needed data that is currently present in meltano.yml.

Important exceptions and deprecations

Setting Description Resolution or Mitigation
settings.env A method of either injecting or sourcing environment variables We support for non-singer plugins. We remove those references used for "sourcing". We remove injection scenarios for Singer plugins unless required (logging an issue with the dev requesting them to support a settings.json - based config). We may also consider a rename to something like as_env_var or mapped_env_var so this is more specifically targeted to injecting env vars rather than its current dual role of injecting and/or sourcing.
settings.env_aliases Same as env but allowing multiple inputs and outputs. Deprecate this. No support.
settings.select Used to create default select patterns for a given tap. Deprecate. This was primarily needed for transform packages (also deprecated) which had specific table and field requirements.
settings.metadata Used similarly to select as a means of overriding selection defaults. Deprecate, same as select metadata.
settings.aliases Alternative names that can be used in meltano.yml and with meltano config set. Keep as-is and provide a list of common/standard aliases to support interop across forks and utilities: e.g. password, user, database_name, schema_name, etc.
settings.value The default value if no other value is provided. Keep, but perhaps rename to default to make the function of this more clear.
settings.placeholder Used to hint to the user what kind of entry is supported Keep (as optional).
settings.description Optional. Used to provide inline context. Keep.
settings.tooltip Optional. Used to provide additional inline context. Keep. Or remove and merge with description.
settings.documentation Optional. Used to provide inline context. Deprecate and merge into description.
settings.value_processor Modify value before loading it from source. Deprecate.
settings.value_post_processor Modify value after loading it from source. Deprecate.
settings.protected Use in combination with value to provide an uneditable default. Deprecate.

Move to a single yaml format in the hub per plugin

Currently we have two yaml files per (discoverable) plugin. This would move us to one-yaml-per-plugin and would allow us to fully deprecate discovery.yml after a certain amount of time has passed.

Alignment of Settings kind

As discussed in Standardize setting types including sensitive a... (#25 - closed), our settings types need to be converted or aligned between Meltano and the hub.

A possible path forward:

  1. Decide in #25 (closed) what the new settings types/kinds will be, probably including a separate "sensitive" boolean flag per setting.
  2. Add support for those new settings types in Meltano.
  3. Deprecate, but do not entirely drop support for the old settings types. (Essentially, those would be aliases until we fully remove them.)
  4. Consider deprecating the older style in Meltano in a 2.0 release.
Edited by AJ Steers