CMAKE: Fix ENABLE_ASAN for MSVC (!14912) · Merge requests · Wireshark Foundation / Wireshark

Problem:

SPECTRE-mitigation (/Qspectre) is enabled by default but this does not allow ASAN (-fsanitize=address) to be enabled because of failing compiler checks for ASAN_FLAG. This problem can also be reproduced with a C++ console application sample using Visual Studio 2019/2022 where it will report linking error related to ASAN library if both the options are enabled.

Cause:

This happens because the ASAN libraries are not present in the path that is used by the linker when SPECTRE-mitigation is enabled.
ASAN libraries are present in VS_TOOLSET_PATH/lib/ but are not present in VS_TOOLSET_PATH/lib/spectre/ which is the path used by SPECTRE-mitigation libraries.
Following links also talk about the same problem. https://github.com/MarkSchofield/WindowsToolchain/issues/72 https://github.com/MarkSchofield/WindowsToolchain/issues/73

Fixes:

First fix is to disable SPECTRE-mitigation when ASAN is requested and is considered a default fix by this change.
Second fix is to add path where ASAN libraries are present and is activated only when ENABLE_ASAN_WITH_SPECTRE is provided on the CMake command line along with ENABLE_ASAN.

Note:

Both the fixes are applicable for Debug and RelWithDebInfo build types only because they provide the necessary debug information required for ASAN to be functional. If ASAN is enabled on Release and MinSizeRel build types too, then warning C5072 gets generated causing build failure (because this warning is treated as an error).

Example:

error C2220: the following warning is treated as an error
warning C5072: ASAN enabled without debug information emission. Enable debug info for better ASAN error reporting.

Important:

Checking if ASAN is supported requires SPECTRE-mitigation to be disabled globally impacting all build types. If ASAN is supported, then it is re-enabled selectively depending on the build types.
As per /libpath documentation, path provided to it is used before all the other paths. This causes SPECTRE-mitigated libraries to be skipped if non-mitigated versions are present in the ASAN library path. For example, comsuppw.lib is present in both the paths. If SPECTRE-mitigation is enabled (along with ASAN), and only ASAN library path is added, then comsuppw.lib from ASAN library path is used!

Summary:

For First or Default Fix
- Debug and RelWithDebInfo: SPECTRE-mitigation stays disabled ASAN is enabled
- Release and MinSizeRel: SPECTRE-mitigation is re-enabled ASAN is skipped
For Second Fix
- Debug and RelWithDebInfo: SPECTRE-mitigation is re-enabled ASAN is enabled
- Release and MinSizeRel: SPECTRE-mitigation is re-enabled ASAN is skipped

CMAKE: Fix ENABLE_ASAN for MSVC