Setting stream map property to `null` does not remove it from SCHEMA message

Summary

There was a partial fix for #352 (closed) which removed the null property from RECORD messages, but we are still seeing the property show up in SCHEMA messages which can break downstream in the target. It's expected that the schema also gets transformed.

Steps to reproduce

"stream_maps": {
  "bounces": {
    "INSERTTS": null
  }
}

What is the current bug behavior?

Resulting schema message:

{"type":"SCHEMA","stream":"bounces","schema":{"properties":{"ACCTCODE":{"type":["number","null"]},"OYBACCTCODE":{"type":["number","null"]},"JOBCODE":{"type":["number","null"]},"BATCHCODE":{"type":["number","null"]},"SUBSCRIBERCODE":{"type":["string","null"]},"SUBSCRIBERKEY":{"type":["string","null"]},"EVENTDATE":{"type":["string","null"]},"BOUNCECATCODE":{"type":["number","null"]},"BOUNCECAT":{"type":["string","null"]},"BOUNCESUBCATCODE":{"type":["number","null"]},"BOUNCESUBCAT":{"type":["string","null"]},"BOUNCETYPECODE":{"type":["number","null"]},"BOUNCETYPE":{"type":["string","null"]},"SMTPMSG":{"type":["string","null"]},"SMTPCODE":{"type":["number","null"]},"SMTPBOUNCEREASON":{"type":["string","null"]},"SENT_UUID":{"type":["string","null"]},"APPLICATIONID":{"type":["string","null"]},"ACCOUNTNUMBER":{"type":["string","null"]},"LEADID":{"type":["string","null"]},"INSERTTS":{"format":"date-time","type":["string","null"]}},"type":"object"},"key_properties":[]}

What is the expected correct behavior?

{"type":"SCHEMA","stream":"bounces","schema":{"properties":{"ACCTCODE":{"type":["number","null"]},"OYBACCTCODE":{"type":["number","null"]},"JOBCODE":{"type":["number","null"]},"BATCHCODE":{"type":["number","null"]},"SUBSCRIBERCODE":{"type":["string","null"]},"SUBSCRIBERKEY":{"type":["string","null"]},"EVENTDATE":{"type":["string","null"]},"BOUNCECATCODE":{"type":["number","null"]},"BOUNCECAT":{"type":["string","null"]},"BOUNCESUBCATCODE":{"type":["number","null"]},"BOUNCESUBCAT":{"type":["string","null"]},"BOUNCETYPECODE":{"type":["number","null"]},"BOUNCETYPE":{"type":["string","null"]},"SMTPMSG":{"type":["string","null"]},"SMTPCODE":{"type":["number","null"]},"SMTPBOUNCEREASON":{"type":["string","null"]},"SENT_UUID":{"type":["string","null"]},"APPLICATIONID":{"type":["string","null"]},"ACCOUNTNUMBER":{"type":["string","null"]},"LEADID":{"type":["string","null"]}},"type":"object"},"key_properties":[]}

Relevant logs and/or screenshots

Possible fixes

I was able to fix locally by changing this line that was previously removed: 55b87d41

The property does need to stay in stream_map but must be removed from transformed_schema with transformed_schema["properties"].pop(prop_key, None)

I will try to open a MR in the next few days with this fix!