[Aarch64/SVE] using -msve-vector-bits=scalable compiler option prevents proper sve detection by provided cmake files
Summary
gromacs' build system does not correctly support -msve-vector-bits=scalable
GROMACS version
'main' branch of https://gitlab.com/gromacs/gromacs
Steps to reproduce
Trying to build via spack with -msve-vector-bits=scalable compiler flags enabled. It's doing something similar to
'/vol0005/mdt3/data/ra000020/u10016/benchpark.llvm/test/spack/opt/spack/linux-rhel8-a64fx/clang-17.0.2/cmake-3.27.9-vkn2fc5xpnpm67eh7wkemafpg5i276pu/bin/cmake' '-G' 'Unix Makefiles' '-DCMAKE_INSTALL_PREFIX:STRING=/vol0005/mdt3/data/ra000020/u10016/benchpark.llvm/test/spack/opt/spack/linux-rhel8-a64fx/clang-17.0.2/gromacs-main-qtuvqznslxpsb2z3alhw7ljau2onpquq' '-DCMAKE_BUILD_TYPE:STRING=Release' '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON' '-DCMAKE_INTERPROCEDURAL_OPTIMIZATION:BOOL=OFF' '-DCMAKE_POLICY_DEFAULT_CMP0090:STRING=NEW' '-DCMAKE_FIND_USE_PACKAGE_REGISTRY:BOOL=OFF' '-DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=ON' '-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON' '-DCMAKE_INSTALL_RPATH:STRING=/vol0005/mdt3/data/ra000020/u10016/benchpark.llvm/test/spack/opt/spack/linux-rhel8-a64fx/clang-17.0.2/gromacs-main-qtuvqznslxpsb2z3alhw7ljau2onpquq/lib;/vol0005/mdt3/data/ra000020/u10016/benchpark.llvm/test/spack/opt/spack/linux-rhel8-a64fx/clang-17.0.2/gromacs-main-qtuvqznslxpsb2z3alhw7ljau2onpquq/lib64;/vol0005/mdt3/data/ra000020/u10016/benchpark.llvm/test/spack/opt/spack/linux-rhel8-a64fx/clang-17.0.2/fftw-3.3.10-ruyxev2eudg6ad6dwujs35ebzewbjykk/lib;/opt/FJSVxtclanga/tcsds-mpi-1.2.38/lib64;/opt/FJSVxtclanga/tcsds-ssl2-1.2.38/lib64' '-DCMAKE_PREFIX_PATH:STRING=/vol0005/mdt3/data/ra000020/u10016/benchpark.llvm/test/spack/opt/spack/linux-rhel8-a64fx/clang-17.0.2/fftw-3.3.10-ruyxev2eudg6ad6dwujs35ebzewbjykk;/vol0005/mdt3/data/ra000020/u10016/benchpark.llvm/test/spack/opt/spack/linux-rhel8-a64fx/clang-17.0.2/cmake-3.27.9-vkn2fc5xpnpm67eh7wkemafpg5i276pu;/vol0005/mdt3/data/ra000020/u10016/benchpark.llvm/test/spack/opt/spack/linux-rhel8-a64fx/clang-17.0.2/gmake-4.4.1-mqcjmyns35tdkt3zbpgyt3ixz6fata5o;/opt/FJSVxtclanga/tcsds-ssl2-1.2.38;/opt/FJSVxtclanga/tcsds-mpi-1.2.38' '-DGMX_MPI:BOOL=ON' '-DCMAKE_C_COMPILER=/vol0005/mdt3/data/ra000020/u10016/benchpark.llvm/test/spack/lib/spack/env/clang/clang' '-DCMAKE_CXX_COMPILER=/vol0005/mdt3/data/ra000020/u10016/benchpark.llvm/test/spack/lib/spack/env/clang/clang++' '-DMPI_C_COMPILER=/opt/FJSVxtclanga/tcsds-mpi-1.2.38/bin/mpifcc' '-DMPI_CXX_COMPILER=/opt/FJSVxtclanga/tcsds-mpi-1.2.38/bin/mpiFCC' '-DGMX_INSTALL_LEGACY_API=ON' '-DGMX_HWLOC:BOOL=OFF' '-DGMX_GPU:STRING=OFF' '-DGMX_EXTERNAL_LAPACK:BOOL=ON' '-DGMX_LAPACK_USER=/opt/FJSVxtclanga/tcsds-ssl2-1.2.38/lib64/libfjlapackexsve.so;/opt/FJSVxtclanga/tcsds-ssl2-1.2.38/lib64/libfjomphk.so;/opt/FJSVxtclanga/tcsds-ssl2-1.2.38/lib64/libfjomp.so;/opt/FJSVxtclanga/tcsds-ssl2-1.2.38/lib64/libssl2mtexsve.a;/opt/FJSVxtclanga/tcsds-ssl2-1.2.38/lib64/libssl2mtsve.a;/opt/FJSVxtclanga/tcsds-ssl2-1.2.38/lib64/libfj90i.so;/opt/FJSVxtclanga/tcsds-ssl2-1.2.38/lib64/libfj90fmt_sve.a;/opt/FJSVxtclanga/tcsds-ssl2-1.2.38/lib64/libfj90f.so;/opt/FJSVxtclanga/tcsds-ssl2-1.2.38/lib64/libfjsrcinfo.so;/opt/FJSVxtclanga/tcsds-ssl2-1.2.38/lib64/libfj90rt.so' '-DGMX_EXTERNAL_BLAS:BOOL=ON' '-DGMX_BLAS_USER=/opt/FJSVxtclanga/tcsds-ssl2-1.2.38/lib64/libfjlapackexsve.so;/opt/FJSVxtclanga/tcsds-ssl2-1.2.38/lib64/libfjomphk.so;/opt/FJSVxtclanga/tcsds-ssl2-1.2.38/lib64/libfjomp.so;/opt/FJSVxtclanga/tcsds-ssl2-1.2.38/lib64/libssl2mtexsve.a;/opt/FJSVxtclanga/tcsds-ssl2-1.2.38/lib64/libssl2mtsve.a;/opt/FJSVxtclanga/tcsds-ssl2-1.2.38/lib64/libfj90i.so;/opt/FJSVxtclanga/tcsds-ssl2-1.2.38/lib64/libfj90fmt_sve.a;/opt/FJSVxtclanga/tcsds-ssl2-1.2.38/lib64/libfj90f.so;/opt/FJSVxtclanga/tcsds-ssl2-1.2.38/lib64/libfjsrcinfo.so;/opt/FJSVxtclanga/tcsds-ssl2-1.2.38/lib64/libfj90rt.so' '-DGMX_SIMD=ARM_SVE' '-DGMX_USE_RDTSCP:BOOL=OFF' '-DGMX_OPENMP:BOOL=ON' '-DGMX_CYCLE_SUBCOUNTERS:BOOL=OFF' '-DGMX_FFT_LIBRARY=fftw3' '-DGMX_VERSION_STRING_OF_FORK=spack' '/local/u10016/spack-stage/spack-stage-gromacs-main-qtuvqznslxpsb2z3alhw7ljau2onpquq/spack-src'
but the compiler flags are set 'internally' by the spack wrappers /vol0005/mdt3/data/ra000020/u10016/benchpark.llvm/test/spack/lib/spack/env/clang/clang[++]
and hence -msve-vector-bits=scalable
will be added to the commandline arguments for the real compiler after -msve-vector-bits=512
which is added by the cmake checks. Trying a normal build outside of spack does not expose this issue because the order is reversed. But either way, IMHO the build system should allow for -msve-vector-bits=scalable builds in general and not just test for 128,..,2k, see:
https://gitlab.com/gromacs/gromacs/-/blob/main/cmake/gmxManageSimd.cmake?ref_type=heads#L230
and
https://gitlab.com/gromacs/gromacs/-/blob/main/cmake/gmxSimdFlags.cmake?ref_type=heads#L311
What is the current bug behavior?
-- The C compiler identification is Clang 17.0.2
-- The CXX compiler identification is Clang 17.0.2
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /vol0005/mdt3/data/ra000020/u10016/benchpark.llvm/test/spack/lib/spack/env/clang/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /vol0005/mdt3/data/ra000020/u10016/benchpark.llvm/test/spack/lib/spack/env/clang/clang++ - skipped
<snip>
-- Detected SVE vector length in bytes : 64
-- Detected SVE vector length of 512 bits
-- Performing Test C_msve_vector_bits_512_FLAG_ACCEPTED
-- Performing Test C_msve_vector_bits_512_FLAG_ACCEPTED - Success
-- Performing Test C_msve_vector_bits_512_COMPILE_WORKS
-- Performing Test C_msve_vector_bits_512_COMPILE_WORKS - Failed
-- Flag was accepted, but it did not build test source (this could be due to either the compiler or binutils)
-- Performing Test C_march_armv8_2_a_sve_msve_vector_bits_512_FLAG_ACCEPTED
-- Performing Test C_march_armv8_2_a_sve_msve_vector_bits_512_FLAG_ACCEPTED - Success
-- Performing Test C_march_armv8_2_a_sve_msve_vector_bits_512_COMPILE_WORKS
-- Performing Test C_march_armv8_2_a_sve_msve_vector_bits_512_COMPILE_WORKS - Failed
-- Flag was accepted, but it did not build test source (this could be due to either the compiler or binutils)
-- Performing Test C_march_armv8_2a_sve_msve_vector_bits_512_FLAG_ACCEPTED
-- Performing Test C_march_armv8_2a_sve_msve_vector_bits_512_FLAG_ACCEPTED - Success
-- Performing Test C_march_armv8_2a_sve_msve_vector_bits_512_COMPILE_WORKS
-- Performing Test C_march_armv8_2a_sve_msve_vector_bits_512_COMPILE_WORKS - Failed
-- Flag was accepted, but it did not build test source (this could be due to either the compiler or binutils)
-- Performing Test C_COMPILE_WORKS_WITHOUT_SPECIAL_FLAGS
-- Performing Test C_COMPILE_WORKS_WITHOUT_SPECIAL_FLAGS - Failed
-- Could not find any flag to build test source (this could be due to either the compiler or binutils)
-- Performing Test CXX_msve_vector_bits_512_FLAG_ACCEPTED
-- Performing Test CXX_msve_vector_bits_512_FLAG_ACCEPTED - Success
-- Performing Test CXX_msve_vector_bits_512_COMPILE_WORKS
-- Performing Test CXX_msve_vector_bits_512_COMPILE_WORKS - Failed
-- Flag was accepted, but it did not build test source (this could be due to either the compiler or binutils)
-- Performing Test CXX_march_armv8_2_a_sve_msve_vector_bits_512_FLAG_ACCEPTED
-- Performing Test CXX_march_armv8_2_a_sve_msve_vector_bits_512_FLAG_ACCEPTED - Success
-- Performing Test CXX_march_armv8_2_a_sve_msve_vector_bits_512_COMPILE_WORKS
-- Performing Test CXX_march_armv8_2_a_sve_msve_vector_bits_512_COMPILE_WORKS - Failed
-- Flag was accepted, but it did not build test source (this could be due to either the compiler or binutils)
-- Performing Test CXX_march_armv8_2a_sve_msve_vector_bits_512_FLAG_ACCEPTED
-- Performing Test CXX_march_armv8_2a_sve_msve_vector_bits_512_FLAG_ACCEPTED - Success
-- Performing Test CXX_march_armv8_2a_sve_msve_vector_bits_512_COMPILE_WORKS
-- Performing Test CXX_march_armv8_2a_sve_msve_vector_bits_512_COMPILE_WORKS - Failed
-- Flag was accepted, but it did not build test source (this could be due to either the compiler or binutils)
-- Performing Test CXX_COMPILE_WORKS_WITHOUT_SPECIAL_FLAGS
-- Performing Test CXX_COMPILE_WORKS_WITHOUT_SPECIAL_FLAGS - Failed
-- Could not find any flag to build test source (this could be due to either the compiler or binutils)
CMake Error at cmake/gmxManageSimd.cmake:50 (message):
Cannot find ARM (AArch64) SVE SIMD compiler flag. Use a newer compiler, or
particularly gcc version 10.1 or later, or clang version 14 or later (when
available), or disable SIMD support (slower).
Call Stack (most recent call first):
cmake/gmxManageSimd.cmake:253 (gmx_give_fatal_error_when_simd_support_not_found)
CMakeLists.txt:690 (gmx_manage_simd)
Reproducing outside of cmake/spack is easy with one of the failing cmake test files:
$ cat ../../../a.c
#include <stdbool.h>
#include<arm_sve.h>
typedef svfloat32_t float32_vec_t __attribute__((arm_sve_vector_bits(512)));
/* check the existence of the svdup_n_b32() intrinsic - currently not implemented by LLVM 12 */
svbool_t duplicate(const bool b) { return svdup_n_b32(b); }
int main(){float32_vec_t x = svdup_f32(0.5f); return 0;}
and
$ /home/u10016/project_SPRT/llvm-v17.0.2/compute_node/bin/clang -DC_msve_vector_bits_512_COMPILE_WORKS -msve-vector-bits=scalable -o a.o -c ../../../a.c
../../../a.c:3:50: error: 'arm_sve_vector_bits' is only supported when '-msve-vector-bits=<bits>' is specified with a value of 128, 256, 512, 1024 or 2048.
3 | typedef svfloat32_t float32_vec_t __attribute__((arm_sve_vector_bits(512)));
| ^
1 error generated.
Possible fixes
Improve or bypass gmxManageSimd.cmake and gmxSimdFlags.cmake when user incists on "custom" -msve-vector-bits=
flags