Secure analyzers don't debug CLI commands before they complete

NOTE if you are a user who also would like to see this feature, please UPVOTE 👍 it and comment to help it get prioritized (So it’s raised as part of our sensing mechanisms. Comments ideally should include what you want, how it would help you, what your pain point/frustration is today, and anything else that can help us focus on solving the problem.

Summary

Since #10880 (closed), the Secure analyzers log the output of the CLI commands they execute with the debug log level. Unfortunately, they rely on Cmd.CombinedOutput to do that, and the standard output and standard are not displayed until the command completes or fails. It makes debugging of the long-running commands harder than it previously was.

Steps to reproduce

Enable Dependency Scanning for a npm project, and set the SECURE_LOG_LEVEL to debug. The retire.js analyzer will run npm install, which is likely to take a significant amount of time to execute.

Example Project

https://gitlab.com/gitlab-org/security-products/tests/js-npm/-/jobs/626759031

What is the current bug behavior?

No output until npm install completes.

What is the expected correct behavior?

The output of npm install shows up immediately.

Relevant logs and/or screenshots

Possible fixes

Log the CLI and its argument using log.Debugf, and then redirect the std out and std err if the log level is debug. If this code becomes significantly complex, then it should probably live in common/logutil.

The output of the CLI commands wouldn't appear as blue anymore, and that's probably a good thing.

/cc @dsearles