Support arbitrary key-value pairs in metadata
Summary
According to the singer metadata spec:
A tap is free to write ANY type of metadata they feel is useful for describing fields in the schema, although several reserved keywords exist.
However, our dataclass implementation prevents the addition of arbitrary fields to the metadata by the tap developer. So, something like this is not possible:
{
"metadata" : {
"selected" : true,
"some-other-metadata" : "whatever"
},
"breadcrumb" : ["properties", "some-field-name"]
}
Proposed benefits
Taps can emit informational metadata. This tags can be later leveraged by data catalogs and orchestrators (Metadata view of Meltano (meltano#2611 - closed)).
Proposal details
As suggested above, tap devs interested in adding key-value pairs to their metadata objects would have to subclass _singer.Metadata etc. This is complicated by the use those of classes in Tap and Stream.
To avoid that, we could easily add a tags property that tap developers can fill-in with relevant metadata:
{
"metadata" : {
"selected" : true,
"tags": {
"some-other-metadata" : "whatever"
}
},
"breadcrumb" : ["properties", "some-field-name"]
}
Best reasons not to build
There may be better ways to support the addition of extra metadata fields.