Skip to content

Slow overflow check compiles to movabs;cqo;idiv;test

Summary

I was looking at the disassembly of a program compiled with libeigen, which uses a lot of matrices with one row and many cols (r15=ncols), and I noticed a lot of this pattern:

movabsq $0x7FFFFFFFFFFFFFFF, %rax
cqto
idivq %r15
testq %rax, %rax
jle throw_std_bad_alloc

This was very surprising to me, as idiv is slow, so (as far as I can tell) it would be far more efficient to just

cmpq $0x7FFFFFFFFFFFFFFF, %r15
jae throw_std_bad_alloc

I knew this was coming from Eigen, but I had trouble figuring out where it was from. SirKane on the reverse-engineering Discord helped me track it down to this:

https://gitlab.com/libeigen/eigen/-/blob/1a2bfca8f000286a13877f493a87fee543e13755/Eigen/src/Core/PlainObjectBase.h#L43

bool error = (rows == 0 || cols == 0) ? false : (rows > max_index / cols);

Even on -O3, gcc doesn't optimize this away: https://godbolt.org/z/eE1bEnE6M

Environment

  • Operating System : Linux
  • Architecture : x86_64
  • Eigen Version : git master
  • Compiler Version : GCC trunk
  • Compile Flags : -O3
  • Vector Extension : N/A
Edited by Ivy Huang