Logging Documentation / Best Practices

Could we get logging documentation and best practices?

I spent a few hours diving into this today. It took me that long because I refused to read the advanced logging section in python for some silly reason. But to the point

I believe https://github.com/transferwise/pipelinewise-singer-python/blob/master/singer/logging.conf is where the majority of the logging configuration is coming from.

Maybe we could tell everyone they could specify a LOGGING_CONF_FILE in their environment variables if they need debug logging. With an example or two?

Context:

1. I want debug logging, and I need to turn on debug logging sometimes in production but there's PII data in the logs so I can't output them all the time.
2. I have a custom target with some custom classes that are using logger.debug("yadayada") and I was manually adding the logging configuration to see this data. In production I need a better way than this (LOGGING_CONF_FILE is one way)

Subtopics (curated by AJ)

1. logging.conf: We can document how works, how to set and give examples for users/devs to utilize this.
2. Changing log level for particularly noisy operations in the SDK.
   • For high-cardinality parent-child relationships, the "starting" and "stopping" section of each partition context can be 10+ times per second.
   • For high-cardinality parent-child relationships, Singer INFO metrics also can be noisy if emitted after each stream.
3. Within a Meltano logging.conf, we might want separate control of the SDK vs Meltano itself.
4. Pretty-printed exceptions, ideally togglable.