Add record and schema flattener in Stream Maps
Rebased, replaces !233 (closed)
This MR adds flattening support as an extension to StreamMaps.
Internally:
- All stream map constructors now take a required
flattening_options
param. Flattening can be disabled by passingNone
or by passing aFlatteningOptions
instance withflattening_enabled=False
ormax_level=0
. - Since
SameRecordTransform
is used even when no other stream maps are applied, this implementation addresses all use cases, regardless of whether stream maps are otherwise invoked. - The flattening function is enabled automatically if
max_flattening_level
is detected in settings, and if its value is>=0
. - A
max_flattening_level
ofnull
or0
will be the same as flattening not being enabled. - Just like stream maps, flattening may be performed at any layer: within tap plugins, target plugins, or standalone mappers. (Traditionally this has only been a target feature.)
- If both stream maps and flattening are enabled, the order of operations will always be:
- first, perform stream map transformations
- second, perform flattening operations.
- if a stream map filters out the record, no flattening will be performed (since no record exists to flatten).
For users (updated 2022-01-18):
- Just specify a config with
flattening_enabled
(bool, off by default) - Optionally,
flattening_max_depth
also can be set to any positive, non-zero number to limit to the specified depth (default10
).
Edited by AJ Steers