[FIXED] numpy.reshape breaks when dealing with a view of a big size

Issue

I have an initial NumPy array of size (512,512,100) of type np.float64 and then I use view_as_windows function to get an array of size (499,499,100,64,64). This function returns a view that consumes much less memory than an actual NumPy array. I want to reshape the view to (499*499*100,64,64). Using the regular np.reshape function, it takes too much time trying to reshape the array, and then it breaks due to trying to convert it to an actual NumPy array. I tried the following, and none of them works:

#gs.shape gives (499,499,100,64,64)
gs.shape = (gs.shape[0]*gs.shape[1]*gs.shape[2],64,64,)
gs = np.reshape(gs,(gs.shape[0]*gs.shape[1]*gs.shape[2],64,64,))
gs = gs.reshape(gs.shape[0]*gs.shape[1]*gs.shape[2],64,64,)

What is the correct way to change the view without actually causing memory overload?

Solution

Why do you want to do this reshape?

Your starting array (with a small dtype):

In [100]: arr = np.ones((512,512,100), 'int8')    
In [101]: arr.nbytes, arr.strides
Out[101]: (26214400, (51200, 100, 1))

Using the view_as_windows as ported to the numpy core:

In [102]: new = np.lib.stride_tricks.sliding_window_view(arr,(64,64),(0,1))

In [103]: new.shape, new.strides
Out[103]: ((449, 449, 100, 64, 64), 
          (51200, 100, 1, 51200, 100))

In [105]: new.nbytes/1e9        # the number of elements (memory if copy)
Out[105]: 82.5757696

Note that the arr final dimension, 100, is now 3rd; in the view is still has a single element stride.

Effectively the first dim, 512, has been 'split' between axis 0 and 3; and the 2nd 512, axis (1,4).

To get the shape, (499499100,64,64), strides would have to be (? , 51200, 100). I'm not an expert on as_strided, but have enough experience to say there's no way of reworking the first 3 strides (51200, 100, 1) into one.

Let's remove the size 100 dimension to simplify things a bit.

In [106]: arr = np.ones((512,512), 'int8')
In [107]: arr.strides
Out[107]: (512, 1)

The windowed:

In [108]: new = np.lib.stride_tricks.sliding_window_view(arr,(64,64),(0,1))
In [109]: new.shape, new.strides
Out[109]: ((449, 449, 64, 64), (512, 1, 512, 1))

and its 3d reshape (a copy):

In [110]: new1 = new.reshape(-1,64,64)    
In [112]: new1.shape, new1.strides
Out[112]: ((201601, 64, 64), (4096, 64, 1))

Answered By - hpaulj