Skip to content

Reduce max relative error of prsqrt from 3 to 2 ulps.

Use a different formulation of the Newton-Raphson step for rsqrt. This was measured exhaustively for all floats using AVX in comparison with the exact value computed by MPFR.

Thanks to Solomon Boulos for the suggestion.

Merge request reports

Loading