Skip to content

Log meltano elt output

Currently, we do not log the output of meltano elt runs (or any other cli command for what is worth).

That means that if meltano elt runs as a sub-process (e.g. a user starting an elt run from Meltano UI) or from within an orchestrator like Airflow, the output of the command is lost or it really hard to be found.

Similarly, even if a meltano elt runs from within a terminal, the output of each run is temporarily available in the terminal and can not be accessed later to check if something strange happened. Also, having access to the whole run depends on the terminal's buffer settings and other external parameters.

We should have available the output of each ELT run in a separate log file, so that we can:

  • Access the output from inside Meltano UI and allow the user to see that everything worked as expected or find errors. This is really important so that users can be reassured that everything is setup as expected, investigate what happens behind the scenes or find errors and submit them as bug reports to the Meltano Team.
  • Make available in real time the output of an ELT run to a user of Meltano UI. At the moment, an ELT run starts and the user just waits there without knowing what happens. Having access to the live log will allow them to know what happens and that the process is not stuck for whatever reason.
  • (Support for future updates) Having the logs for each run, adds the ability to process ELT run outputs by Meltano. That means scanning the logs for errors, identifying wrong credentials or configurations and providing hints to the user with respect to what went wrong.

The approach that I propose we should use in the first iteration of this implementation is:

  1. Capture all [stdout, stderr] output and redirect it to both the terminal and a log file
  2. Logs will be placed under .meltano/run/logs/
  3. Each meltano elt will be logged using its unique job_id in a file named .meltano/run/logs/elt_{job_id}.log
  4. We will keep only the last execution with the same job_id (i.e. open the file in write, not append mode) to protect the logs from getting too large. The idea is that you only need the logs from the last execution each time to successfully support the requirements described above.