Resolve "Emit structured (JSON) logs"

Merge Request Checklist

• Link to an issue
• Include the proposed fix or feature
• Include and update tests for the modified code
• Include a documentation change
• Add a CHANGELOG.md entry in the Unreleased section for any user-facing changes
• Mention or assign a maintainer

Closes #2772 (closed)

How this works

1. We'll look for a logging.yaml in the cwd by default. You can control the config name via --log-config. If a config is found its loaded and used.
2. As-is - when ELT is run we still write to the logs for the run to a dedicated path (i.e. .meltano/logs/elt/job_id/run_id/elt.log). We'll write there in addition to whatever config is present in a logging.yaml as we're just temporarily injecting an additional handler.
3. If no logging config is found, we'll setup a default. When the default config is active, --log-level can be used to control the log level.

To get this setup - I used the structlog package. There's a few implementation paths we could have taken, I opted for implementing rendering using structlog based formatters within logging. details here since it seemed to tick all the boxes for us, but mainly:

• we can convert to explicit structured logging calls as we go along, similar to how we've been adding type hints
• existing logging calls get converted
• easy integration with logging.yaml/dictConfig setup for end-users

What renderers and handlers are supported

Theoretically, all handlers as mentioned in the logging.handlers docs. On top of those handlers, you can also team them up with renderers to produce:

• colorized structured console output
• un-colorized structured console output
• json formatted output
• key=value formatted
• any plain text custom formats

Work remaining

• tests
• docs (for contributors)
• bulk refactor away from vanilla logging calls to struct log calls ? (IMHO probably best to treat this like type hints, and refactors as we come across them, but might be worth proactively updating meltano/core/runner/singer.py, meltano/core/plugin/singer/tap.py, and meltano/core/plugin/singer/target.py)

Sample Screenshots

Two sample logs, one with a tap failure, one successful. Including screenshots to illustrate what the new colored console output can look like. Output colors are dependent on the users personal console theme and shell - basic case you'd only get colorized log levels: