Test matrix_cwise_3: undefined behaviour

Summary

Test matrix_cwise_3 has undefined behaviour. While it seems to go unnoticed on x86, on s390x with clang it leads to test failure. On s390x with gcc it goes unnoticed too.

Environment

  • Operating System : Linux
  • Architecture : x64
  • Eigen Version : master branch (currently, commit d81aa18f)
  • Compiler Version : clang 20.1.3
  • Compile Flags : -fsanitize=undefined -fsanitize-trap=undefined
  • Vector Extension : none AFAIK

Minimal Example

Dockerfile:

FROM fedora:42

RUN dnf update -y && dnf install -y \
        git \
        cmake \
        clang \
        clang++ \
        gdb

RUN git clone https://gitlab.com/libeigen/eigen/

RUN mkdir eigen/build && \
        cd eigen/build && \
        CC=clang CXX=clang++ cmake .. -DBUILD_TESTING=ON \
        -DCMAKE_BUILD_TYPE=Debug -DEIGEN_TEST_SSE2=ON \
        -DEIGEN_TEST_SSE3=ON -DEIGEN_TEST_SSSE3=ON \
        -DEIGEN_TEST_SSE4_1=ON -DEIGEN_TEST_SSE4_2=ON \
        -DEIGEN_TEST_AVX=OFF -DEIGEN_TEST_AVX2=OFF \
        -DEIGEN_TEST_FMA=ON -DEIGEN_TEST_AVX512=OFF \
        -DCMAKE_CXX_STANDARD=17 \
        -DEIGEN_TEST_CUSTOM_CXX_FLAGS="-fsanitize=undefined -fsanitize-trap=undefined" \
        -DEIGEN_TEST_CUSTOM_LINKER_FLAGS="-fsanitize=undefined -fsanitize-trap=undefined" && \
        make matrix_cwise_3

Steps to reproduce

  1. Build docker image: docker build -f eigen_matrix_cwise.Dockerfile -t eigen-ub-1 .
  2. Run it: docker run -it --rm eigen-ub-1
  3. Run failing test matrix_cwise_3, for example with seed 1748428735:
# cd eigen/build
# gdb --args ./test/matrix_cwise_3 s1748428735
(gdb) run
Enable debuginfod for this session? (y or [n]) n

What is the current bug behavior?

Undefined behaviour is hit. On s390x with clang this leads to matrix_cwise_3 test failing. On other platforms it seems to go unnoticed.

Program received signal SIGILL, Illegal instruction.
0x0000000000448a7f in Eigen::internal::sqrt_impl<signed char>::run (x=@0x3d643f83: -128 '\200') at /eigen/Eigen/src/Core/MathFunctions.h:288
288         return sqrt(x);
Missing rpms, try: dnf --enablerepo='*debug*' install libstdc++-debuginfo-15.1.1-2.fc42.x86_64 glibc-debuginfo-2.41-5.fc42.x86_64 libgcc-debuginfo-15.1.1-2.fc42.x86_64
(gdb) bt
#0  0x0000000000448a7f in Eigen::internal::sqrt_impl<signed char>::run (x=@0x3d643f83: -128 '\200') at /eigen/Eigen/src/Core/MathFunctions.h:288
#1  Eigen::numext::sqrt<signed char> (x=@0x3d643f83: -128 '\200') at /eigen/Eigen/src/Core/MathFunctions.h:1349
#2  test_cwise_real<Eigen::Matrix<signed char, -1, -1, 0, -1, -1> >(Eigen::Matrix<signed char, -1, -1, 0, -1, -1> const&)::{lambda(signed char const&)#7}::operator()(signed char const&) const (
    this=0x7ffc06028e97, x=@0x3d643f83: -128 '\200') at /eigen/test/matrix_cwise.cpp:73
#3  0x000000000041df62 in cwise_ref<Eigen::Matrix<signed char, -1, -1, 0, -1, -1>, test_cwise_real<Eigen::Matrix<signed char, -1, -1, 0, -1, -1> >(Eigen::Matrix<signed char, -1, -1, 0, -1, -1> const&)::{lambda(signed char const&)#7}, Eigen::Matrix<signed char, -1, -1, 0, -1, -1> >(Eigen::Matrix<signed char, -1, -1, 0, -1, -1> const&, test_cwise_real<Eigen::Matrix<signed char, -1, -1, 0, -1, -1> >(Eigen::Matrix<signed char, -1, -1, 0, -1, -1> const&)::{lambda(signed char const&)#7}) (m=..., f=...) at /eigen/test/matrix_cwise.cpp:31
#4  0x00000000004073ab in test_cwise_real<Eigen::Matrix<signed char, -1, -1, 0, -1, -1> > (m=...) at /eigen/test/matrix_cwise.cpp:73
#5  0x0000000000402821 in test_matrix_cwise () at /eigen/test/matrix_cwise.cpp:293
#6  0x0000000000406a50 in Eigen::EigenTest::operator() (this=0xfb5418 <test_handler_matrix_cwise>) at /eigen/test/main.h:202
#7  0x0000000000402677 in main (argc=2, argv=0x7ffc0602aab8) at /eigen/test/main.h:913

(gdb) frame 4
#4  0x00000000004073ab in test_cwise_real<Eigen::Matrix<signed char, -1, -1, 0, -1, -1> > (m=...) at /eigen/test/matrix_cwise.cpp:73
73        VERIFY_IS_CWISE_APPROX(m2.cwiseSqrt(), cwise_ref(m2, [](const Scalar& x) { return Eigen::numext::sqrt(x); }));
(gdb) l
68        }
69        VERIFY_IS_CWISE_APPROX(m2.cwiseInverse(), cwise_ref(m2, [](const Scalar& x) { return Scalar(Scalar(1) / x); }));
70        VERIFY_IS_CWISE_APPROX(m1.cwiseArg(), cwise_ref(m1, [](const Scalar& x) { return Eigen::numext::arg(x); }));
71        // Only take sqrt of positive values.
72        m2 = m1.cwiseAbs();
73        VERIFY_IS_CWISE_APPROX(m2.cwiseSqrt(), cwise_ref(m2, [](const Scalar& x) { return Eigen::numext::sqrt(x); }));
74        // Only find Square/Abs2 of +/- sqrt values so we don't overflow.
75        m2 = m2.cwiseSqrt().array() * m1.cwiseSign().array();
76        VERIFY_IS_CWISE_APPROX(m2.cwiseAbs2(), cwise_ref(m2, [](const Scalar& x) { return Eigen::numext::abs2(x); }));
77        VERIFY_IS_CWISE_APPROX(m2.cwiseSquare(), cwise_ref(m2, [](const Scalar& x) { return Scalar(x * x); }));
(gdb) print m1.m_storage.m_data[16 * 20 + 3]
$10 = -128 '\200'
(gdb) print m2.m_storage.m_data[16 * 20 + 3]
$11 = -128 '\200'

It looks like that matrix m1 of signed char has one value of -128. While running cwiseAbs() function, this value is transformed into 128, but 128 cannot fit into signed char, and it looks likeit get's stored as -128 due to implementation-defined result in such case.

Then sqrt(-128) is called, and result is NAN, which cannot be stored into signed char, and thus it is undefined behaviour.

What is the expected correct behavior?

No UB. There are a couple of ways to prevent that. For example, process special case of value == std::numeric_limits<T>::min() when T is signed integer (int8_t, int16_t, and so on...) in abs calls, or specially process values < 0 for signed integer types in sqrt call.

Relevant logs

Reproduces in docker with latest fedora and recent clang:

# cat /etc/os-release 
NAME="Fedora Linux"
VERSION="42 (Container Image)"
RELEASE_TYPE=stable
ID=fedora
VERSION_ID=42
VERSION_CODENAME=""
PLATFORM_ID="platform:f42"
PRETTY_NAME="Fedora Linux 42 (Container Image)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:42"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f42/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=42
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=42
SUPPORT_END=2026-05-13
VARIANT="Container Image"
VARIANT_ID=container

# clang++ --version
clang version 20.1.3 (Fedora 20.1.3-1.fc42)
Target: x86_64-redhat-linux-gnu
Thread model: posix
InstalledDir: /usr/lib64/llvm20/bin
Configuration file: /etc/clang/x86_64-redhat-linux-gnu-clang++.cfg

Warning Messages

Benchmark scripts and results

Anything else that might help

  • Have a plan to fix this issue.