[FIXED] How to workaround occasional TypeError while using numpy.bitwise_xor with large numbers?

Issue

While analyzing data containing large numbers (256 bits each to be exact), I've stumbled upon a different behavior of numpy.bitwise_xor operation. Depending on the input integers values, the function returns a number (as expected) or a TypeError.

Testing the numpy.bitwise_xor function with two sample data:

-  4035225266123964411 (dec) = 0011011111111111111111111111111111111111111111111111111111111011 (bin)
- 18446744073709551614 (dec) = 1111111111111111111111111111111111111111111111111111111111111110 (bin)

If both arguments are the same (either one of the above numbers), the function returns 0 (as expected). But if I try to xor the two numbers above with each other, I get the error: TypeError: ufunc 'bitwise_xor' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''.

import numpy as np

np.bitwise_xor(4035225266123964411, 4035225266123964411)
# returns: 0
np.bitwise_xor(18446744073709551614, 18446744073709551614)
# returns: 0
np.bitwise_xor(4035225266123964411, 18446744073709551614)
# returns: TypeError

Why the numpy.bitwise_xor function reports an error with casting in one case, and has no problem with that in the other case? Do you have any suggestion how to workaround this issue and xor the large (up 256 bits) numbers in python (ideally without splitting the numbers to 32/64 bit chunks)?

Solution

Unless explicitly told otherwise, numpy uses its own datatypes to contain values upon which it's operating. 4035225266123964411 is converted to np.int64. 18446744073709551614 is too large to fit in the biggest native numpy integer type (np.int64) so its type is object (formerly np.object). Remember that Python integers are objects.

In the first case you have two np.int64 values. No problem with that.

np.bitwise_xor(4035225266123964411, 4035225266123964411)
# np.bitwise_xor(np.int64, np.int64)

In the second case you have two object values. No problem as long as .__xor__ is implemented for those objects - which it is for int.

np.bitwise_xor(18446744073709551614, 18446744073709551614)
# np.bitwise_xor(object, object)

The third case is problematic because you have two values of differing types,np.int64 and object, which aren't implicitly compatible with each other.

np.bitwise_xor(4035225266123964411, 18446744073709551614)
# np.bitwise_xor(np.int64, object) <-- unable to safely coerce into the same type

You can manually coerce them to the same type by specifying the dtype argument.

np.bitwise_xor(4035225266123964411, 18446744073709551614, dtype=np.int64)
# bad; 18446744073709551614 is too large and overflows

# -- or --

np.bitwise_xor(4035225266123964411, 18446744073709551614, dtype=object)
# good; both numbers are treated as type "object" making the result
# equivalent to the native Python operation:
#    4035225266123964411 ^ 18446744073709551614

Answered By - Woodford