Siphon: allow skipping columns
Extend the TableMappingConfig with column transform configuration (we might have different kinds of transformations later):
table_mapping:
- table: issues
schema: public
stream: "users"
column_transformations:
- name: "api_key"
transform: "ignore" # I'm open to suggestions about the naming
When parsing the LR event and we see the api_key column, we would ignore/skip the column.
Reasoning: this is a security feature to prevent pushing keys and secrets to the pub-sub system.
Edited by Adam Hegyi