wslog assumes stderr and stdout exist

Summary

Building and debugging with Visual Studio 2019, Wireshark will assert during ws_log_init at line 833/844 and raise an exception. stdout and stderr are null? and return -2 when passed to fileno() so deep inside g_log_writer_supports_color() an assert occurs: "g_log_writer_supports_color: assertion 'output_fd >= 0' failed"

Steps to reproduce

Follow Windows build instructions with wireshark commit 47a1b0f9 ... cmake -G "Visual Studio 16 2019" -A x64 ..\wireshark

msbuild /m /p:Configuration=RelWithDebInfo Wireshark.sln

Open Wireshark.sln from the developer command prompt

Confirm Executables>wireshark (right click) is "set as Startup project".

Confirm Solution Configuration is set to RelWithDebInfo.

Attempt to execute Wireshark with the Visual studio debugger (F5 typically).

--assert at ws_log_init at line 833

Note, this only occurs if launched with visual studios debugging. If the same .exe is launched with a console or double click it seems to work normally.

What is the current bug behavior?

Because the Visual Studio project (wireshark) is configured with a default linker flag /SUBSYSTEM:WINDOWS, when visual studio launches the program, the file descriptors for stdout and stderr are not valid. This is a quirk of visual studio. So in ws_log_init at line 833, the fileno(stdout) call returns -2 and then g_log_writer_supports_color(-2) fails an assert inside glib here's the stack trace:

[Inline Frame] glib-2.0-0.dll!_g_log_abort(int) Line 557

glib-2.0-0.dll!g_logv(const char * log_domain, GLogLevelFlags log_level, const char * format, char * args) Line 1414

glib-2.0-0.dll!g_log(const char * log_domain, GLogLevelFlags log_level, const char * format, ...) Line 1458

[Inline Frame] glib-2.0-0.dll!g_return_if_fail_warning(const char *) Line 2942

glib-2.0-0.dll!g_log_writer_supports_color(int output_fd) Line 2092

libwsutil.dll!ws_log_init(const char * progname, void(*)(const char *, char *) vcmdarg_err) Line 834

Wireshark.exe!main(int argc, char * * qt_argv) Line 511

Wireshark.exe!WinMain(HINSTANCE__ * __formal, HINSTANCE__ * __formal, char * __formal, int __formal) Line 97

The error message (from debug variables) is: "g_log_writer_supports_color: assertion 'output_fd >= 0' failed"

What is the expected correct behavior?

In this particular case I'm not certain, but the program shouldn't crash. There are a few solutions but I'm not familiar enough with all of the use cases to know which is best.

1. Check fileno(stdout) and fileno(stderr) in ws_log_init before calling g_log_writer_supports_color().
2. Configure cmake files to set the linker flag /SUBSYSTEM:CONSOLE so that the program is always launched with a console. set_target_properties(${PROJECT_NAME} PROPERTIES LINK_FLAGS /SUBSYSTEM:CONSOLE )
3. Hide lines calling g_log_writer_supports_color behind #ifdef _WIN32

Sample capture file

Not applicable.

Relevant logs and/or screenshots

Build information

-- Selecting Windows SDK version 10.0.20348.0 to target Windows 10.0.19042. -- Generating build using CMake 3.24.0-rc2

Other

I had been on 4.0.0 release tag until recently jumping back to master and performing a clean re-build (removed libraries as well). It appears that this commit 2dd07bc5 removed some glib version guards that had been isolating me from this issue.