Fix measure tool glitch at 180 degrees

Fix for #2243 (closed)

In the picture below you can see the smallest value I was able to get for the denominator without holding ctrl (see added comments in the fix). So the code now checks for a range of values that are much smaller than this such that no instabilities occur.