Record flattening for targets

For targets which land data in a columnar format, 'data flattening' has become a common problem for these types of targets. We can implement a Sink.flatten_record(record, max_level) and Sink.flatten_schema(schema, max_level) method which would take a nested structure and output a columnar one.

For instance, { "col_a": { "subcol_1": 1, "subcol_2": 2 } would become { "col_a__subcol_1": 1, "col_a__subcol_2": 2 }.

  1. Tap developers should be able to leverage flattening behavior automatically unless they want to disable it.
    • For example, a target-jsonl destination would have this disabled, whereas target-snowflake might have it on by default and configurable by the user.
    • Unclear if this should be opt-in or opt-out on the developer's side.
  2. Assuming flattening is allowed by the target, the user should be able to disable or customize flattening with a config option like max_flattening_level, where max_flattening_level = 0 disables the flattening behavior altogether.