Test matrix_cwise_3: undefined behaviour
<!--
Please read this!
Before opening a new issue, make sure to search for keywords in the issues
filtered by "bug::confirmed" or "bug::unconfirmed" and "bugzilla" label:
- https://gitlab.com/libeigen/eigen/-/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=bug%3A%3Aconfirmed
- https://gitlab.com/libeigen/eigen/-/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=bug%3A%3Aunconfirmed
- https://gitlab.com/libeigen/eigen/-/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=bugzilla
and verify the issue you're about to submit isn't a duplicate. -->
### Summary
<!-- Summarize the bug encountered concisely. -->
Test `matrix_cwise_3` has undefined behaviour. While it seems to go unnoticed on x86, on s390x with clang it leads to test failure. On s390x with gcc it goes unnoticed too.
### Environment
<!-- Please provide your development environment here -->
- **Operating System** : Linux
- **Architecture** : x64
- **Eigen Version** : master branch (currently, commit d81aa18f4)
- **Compiler Version** : clang 20.1.3
- **Compile Flags** : -fsanitize=undefined -fsanitize-trap=undefined
- **Vector Extension** : none AFAIK
### Minimal Example
<!-- If possible, please create a minimal example here that exhibits the problematic behavior.
You can also link to [godbolt](https://godbolt.org). But please note that you need to click
the "Share" button in the top right-hand corner of the godbolt page where you reproduce the sample
code to get the share link instead of in your browser address bar.
You can read [the guidelines on stackoverflow](https://stackoverflow.com/help/minimal-reproducible-example)
on how to create a good minimal example. -->
Dockerfile:
```
FROM fedora:42
RUN dnf update -y && dnf install -y \
git \
cmake \
clang \
clang++ \
gdb
RUN git clone https://gitlab.com/libeigen/eigen/
RUN mkdir eigen/build && \
cd eigen/build && \
CC=clang CXX=clang++ cmake .. -DBUILD_TESTING=ON \
-DCMAKE_BUILD_TYPE=Debug -DEIGEN_TEST_SSE2=ON \
-DEIGEN_TEST_SSE3=ON -DEIGEN_TEST_SSSE3=ON \
-DEIGEN_TEST_SSE4_1=ON -DEIGEN_TEST_SSE4_2=ON \
-DEIGEN_TEST_AVX=OFF -DEIGEN_TEST_AVX2=OFF \
-DEIGEN_TEST_FMA=ON -DEIGEN_TEST_AVX512=OFF \
-DCMAKE_CXX_STANDARD=17 \
-DEIGEN_TEST_CUSTOM_CXX_FLAGS="-fsanitize=undefined -fsanitize-trap=undefined" \
-DEIGEN_TEST_CUSTOM_LINKER_FLAGS="-fsanitize=undefined -fsanitize-trap=undefined" && \
make matrix_cwise_3
```
### Steps to reproduce
<!-- Describe how one can reproduce the issue - this is very important. Please use an ordered list. -->
1. Build docker image: `docker build -f eigen_matrix_cwise.Dockerfile -t eigen-ub-1 .`
2. Run it: `docker run -it --rm eigen-ub-1`
3. Run failing test `matrix_cwise_3`, for example with seed `1748428735`:
```
# cd eigen/build
# gdb --args ./test/matrix_cwise_3 s1748428735
(gdb) run
Enable debuginfod for this session? (y or [n]) n
```
### What is the current *bug* behavior?
Undefined behaviour is hit. On s390x with clang this leads to `matrix_cwise_3` test failing. On other platforms it seems to go unnoticed.
```
Program received signal SIGILL, Illegal instruction.
0x0000000000448a7f in Eigen::internal::sqrt_impl<signed char>::run (x=@0x3d643f83: -128 '\200') at /eigen/Eigen/src/Core/MathFunctions.h:288
288 return sqrt(x);
Missing rpms, try: dnf --enablerepo='*debug*' install libstdc++-debuginfo-15.1.1-2.fc42.x86_64 glibc-debuginfo-2.41-5.fc42.x86_64 libgcc-debuginfo-15.1.1-2.fc42.x86_64
(gdb) bt
#0 0x0000000000448a7f in Eigen::internal::sqrt_impl<signed char>::run (x=@0x3d643f83: -128 '\200') at /eigen/Eigen/src/Core/MathFunctions.h:288
#1 Eigen::numext::sqrt<signed char> (x=@0x3d643f83: -128 '\200') at /eigen/Eigen/src/Core/MathFunctions.h:1349
#2 test_cwise_real<Eigen::Matrix<signed char, -1, -1, 0, -1, -1> >(Eigen::Matrix<signed char, -1, -1, 0, -1, -1> const&)::{lambda(signed char const&)#7}::operator()(signed char const&) const (
this=0x7ffc06028e97, x=@0x3d643f83: -128 '\200') at /eigen/test/matrix_cwise.cpp:73
#3 0x000000000041df62 in cwise_ref<Eigen::Matrix<signed char, -1, -1, 0, -1, -1>, test_cwise_real<Eigen::Matrix<signed char, -1, -1, 0, -1, -1> >(Eigen::Matrix<signed char, -1, -1, 0, -1, -1> const&)::{lambda(signed char const&)#7}, Eigen::Matrix<signed char, -1, -1, 0, -1, -1> >(Eigen::Matrix<signed char, -1, -1, 0, -1, -1> const&, test_cwise_real<Eigen::Matrix<signed char, -1, -1, 0, -1, -1> >(Eigen::Matrix<signed char, -1, -1, 0, -1, -1> const&)::{lambda(signed char const&)#7}) (m=..., f=...) at /eigen/test/matrix_cwise.cpp:31
#4 0x00000000004073ab in test_cwise_real<Eigen::Matrix<signed char, -1, -1, 0, -1, -1> > (m=...) at /eigen/test/matrix_cwise.cpp:73
#5 0x0000000000402821 in test_matrix_cwise () at /eigen/test/matrix_cwise.cpp:293
#6 0x0000000000406a50 in Eigen::EigenTest::operator() (this=0xfb5418 <test_handler_matrix_cwise>) at /eigen/test/main.h:202
#7 0x0000000000402677 in main (argc=2, argv=0x7ffc0602aab8) at /eigen/test/main.h:913
(gdb) frame 4
#4 0x00000000004073ab in test_cwise_real<Eigen::Matrix<signed char, -1, -1, 0, -1, -1> > (m=...) at /eigen/test/matrix_cwise.cpp:73
73 VERIFY_IS_CWISE_APPROX(m2.cwiseSqrt(), cwise_ref(m2, [](const Scalar& x) { return Eigen::numext::sqrt(x); }));
(gdb) l
68 }
69 VERIFY_IS_CWISE_APPROX(m2.cwiseInverse(), cwise_ref(m2, [](const Scalar& x) { return Scalar(Scalar(1) / x); }));
70 VERIFY_IS_CWISE_APPROX(m1.cwiseArg(), cwise_ref(m1, [](const Scalar& x) { return Eigen::numext::arg(x); }));
71 // Only take sqrt of positive values.
72 m2 = m1.cwiseAbs();
73 VERIFY_IS_CWISE_APPROX(m2.cwiseSqrt(), cwise_ref(m2, [](const Scalar& x) { return Eigen::numext::sqrt(x); }));
74 // Only find Square/Abs2 of +/- sqrt values so we don't overflow.
75 m2 = m2.cwiseSqrt().array() * m1.cwiseSign().array();
76 VERIFY_IS_CWISE_APPROX(m2.cwiseAbs2(), cwise_ref(m2, [](const Scalar& x) { return Eigen::numext::abs2(x); }));
77 VERIFY_IS_CWISE_APPROX(m2.cwiseSquare(), cwise_ref(m2, [](const Scalar& x) { return Scalar(x * x); }));
(gdb) print m1.m_storage.m_data[16 * 20 + 3]
$10 = -128 '\200'
(gdb) print m2.m_storage.m_data[16 * 20 + 3]
$11 = -128 '\200'
```
It looks like that matrix `m1` of `signed char` has one value of `-128`. While running `cwiseAbs()` function, this value is transformed into `128`, but `128` cannot fit into `signed char`, and it looks likeit get's stored as `-128` due to implementation-defined result in such case.
Then `sqrt(-128)` is called, and result is `NAN`, which cannot be stored into `signed char`, and thus it is undefined behaviour.
### What is the expected *correct* behavior?
<!-- Describe what you should see instead. -->
No UB. There are a couple of ways to prevent that. For example, process special case of `value == std::numeric_limits<T>::min()` when `T` is signed integer (int8_t, int16_t, and so on...) in `abs` calls, or specially process values < 0 for signed integer types in sqrt call.
### Relevant logs
<!-- Add relevant code snippets or program output within blocks marked by " ``` " -->
Reproduces in docker with latest fedora and recent clang:
```
# cat /etc/os-release
NAME="Fedora Linux"
VERSION="42 (Container Image)"
RELEASE_TYPE=stable
ID=fedora
VERSION_ID=42
VERSION_CODENAME=""
PLATFORM_ID="platform:f42"
PRETTY_NAME="Fedora Linux 42 (Container Image)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:42"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f42/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=42
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=42
SUPPORT_END=2026-05-13
VARIANT="Container Image"
VARIANT_ID=container
# clang++ --version
clang version 20.1.3 (Fedora 20.1.3-1.fc42)
Target: x86_64-redhat-linux-gnu
Thread model: posix
InstalledDir: /usr/lib64/llvm20/bin
Configuration file: /etc/clang/x86_64-redhat-linux-gnu-clang++.cfg
```
<!-- OPTIONAL: remove this section if you are not reporting a compilation warning issue.-->
### Warning Messages
<!-- Show us the warning messages you got! -->
<!-- OPTIONAL: remove this section if you are not reporting a performance issue. -->
### Benchmark scripts and results
<!-- Please share any benchmark scripts - either standalone, or using [Google Benchmark](https://github.com/google/benchmark). -->
### Anything else that might help
<!-- It will be better to provide us more information to help narrow down the cause.
Including but not limited to the following:
- lines of code that might help us diagnose the problem.
- potential ways to address the issue.
- last known working/first broken version (release number or commit hash). -->
- [ ] Have a plan to fix this issue.
issue