[FIXED] Numpy overflow warning when when subtracting 1 from 2**31 but not from numpy.power(2,31) when storing the results in int32

Issue

This might be a little nitty gritty, but I would like to know this behavior of Numpy in more detail (numpy version 1.26.3).

I encountered this when programming a simple Park and Miller random number generator. It involves calculating the modulus with 2^31-1 and just happened to define it exactly like that:

m = numpy.power(2,31)-1

and got a warning RuntimeWarning: overflow encountered in scalar subtract m = numpy.power(2,31)-1. I was slightly surprised at first, but I quickly realized that numpy works with int32 by default, even in 64-bit machines. But that got me wondering why there was no error on the power, which overflows the signed 32-bit integer. The result is still correct:

print(type(m),m)

<class 'numpy.int32'> 2147483647

Direct storage of 2147483648 to m gives overflow error OverflowError: Python int too large to convert to C long.

This worked well; I got a warning now that I was storing 2^31-1. If I had instead stored 2^31, I would have got a wrong answer without warning:

test = np.power(2,31)
print(type(test),test)

<class 'numpy.int32'> -2147483648

Instead of the correct answer, 2147483648. Is there a guideline on when to carefully check for this kind of behavior with Numpy?

Solution

As I suspected, it was a duplicate question. Numpy does not do overflow detection on arrays. Apparently, that needs hardware support to keep array operations fast enough for large arrays, and current hardware generations don't support it.

See

https://github.com/numpy/numpy/issues/8987 and

Using numpy to square value gives negative number

as posted by https://stackoverflow.com/users/901925/hpaulj as a comment.

Answered By - PSm