Record flattening for targets
For targets which land data in a columnar format, 'data flattening' has become a common problem for these types of targets. We can implement a Sink.flatten_record(record, max_level)
and Sink.flatten_schema(schema, max_level)
method which would take a nested structure and output a columnar one.
For instance, { "col_a": { "subcol_1": 1, "subcol_2": 2 }
would become { "col_a__subcol_1": 1, "col_a__subcol_2": 2 }
.
- Tap developers should be able to leverage flattening behavior automatically unless they want to disable it.
- For example, a
target-jsonl
destination would have this disabled, whereastarget-snowflake
might have it on by default and configurable by the user. - Unclear if this should be opt-in or opt-out on the developer's side.
- For example, a
- Assuming flattening is allowed by the target, the user should be able to disable or customize flattening with a config option like
max_flattening_level
, wheremax_flattening_level = 0
disables the flattening behavior altogether.