Skip to content

Add record and schema flattener in Stream Maps

AJ Steers requested to merge aj-flattener-2 into main

Rebased, replaces !233 (closed)

This MR adds flattening support as an extension to StreamMaps.

Internally:

  • All stream map constructors now take a required flattening_options param. Flattening can be disabled by passing None or by passing a FlatteningOptions instance with flattening_enabled=False or max_level=0.
  • Since SameRecordTransform is used even when no other stream maps are applied, this implementation addresses all use cases, regardless of whether stream maps are otherwise invoked.
  • The flattening function is enabled automatically if max_flattening_level is detected in settings, and if its value is >=0.
  • A max_flattening_level of null or 0 will be the same as flattening not being enabled.
  • Just like stream maps, flattening may be performed at any layer: within tap plugins, target plugins, or standalone mappers. (Traditionally this has only been a target feature.)
  • If both stream maps and flattening are enabled, the order of operations will always be:
    • first, perform stream map transformations
    • second, perform flattening operations.
    • if a stream map filters out the record, no flattening will be performed (since no record exists to flatten).

For users (updated 2022-01-18):

  • Just specify a config with flattening_enabled (bool, off by default)
  • Optionally, flattening_max_depth also can be set to any positive, non-zero number to limit to the specified depth (default 10).
Edited by AJ Steers

Merge request reports