[FIXED] Build a list of array without ',' with sliding window

Issue

I have an array a as follow:

import numpy as np
a= np.array([[1, 3, 5, 7, 8, 7, 1],
   [11, 13, 51, 17, 18, 17, 10]])

I want to build a list of that array with a sliding window. Here is the output that I want:

I have using the following code, however it does not provide the output which I want:

lag           = 3
out = []
for i in range(2):
    eachrow  =[]
    for col in range(a.shape[1]-lag):
        X_row = []
        xtmp = a[i, col:col+lag]
        X_row.append(xtmp)
        ytmp = a[i, col+lag]  
        X_row.append(ytmp)
        eachrow.append(X_row)
    out.append(eachrow)

Any help appreciated. Thanks.

Solution

Your code produces:

In [3]: out
Out[3]: 
[[[array([1, 3, 5]), 7],
  [array([3, 5, 7]), 8],
  [array([5, 7, 8]), 7],
  [array([7, 8, 7]), 1]],
 [[array([11, 13, 51]), 17],
  [array([13, 51, 17]), 18],
  [array([51, 17, 18]), 17],
  [array([17, 18, 17]), 10]]]

That's a list of length 2. Within that lists.

If we make an array from that - with object dtype, we get a 3d array, where some elements are arrays, and some are integers:

In [6]: arr = np.array(out, object)
In [7]: arr
Out[7]: 
array([[[array([1, 3, 5]), 7],
        [array([3, 5, 7]), 8],
        [array([5, 7, 8]), 7],
        [array([7, 8, 7]), 1]],

       [[array([11, 13, 51]), 17],
        [array([13, 51, 17]), 18],
        [array([51, 17, 18]), 17],
        [array([17, 18, 17]), 10]]], dtype=object)
In [8]: arr.shape
Out[8]: (2, 4, 2)

Change one line of your code to

     X_row.append(np.array([ytmp]))

In [11]: np.array(out,object)
Out[11]: 
array([[[array([1, 3, 5]), array([7])],
        [array([3, 5, 7]), array([8])],
        [array([5, 7, 8]), array([7])],
        [array([7, 8, 7]), array([1])]],

       [[array([11, 13, 51]), array([17])],
        [array([13, 51, 17]), array([18])],
        [array([51, 17, 18]), array([17])],
        [array([17, 18, 17]), array([10])]]], dtype=object)

or displayed with the str/print array formatting:

In [12]: print(_)
[[[array([1, 3, 5]) array([7])]
  [array([3, 5, 7]) array([8])]
  [array([5, 7, 8]) array([7])]
  [array([7, 8, 7]) array([1])]]

 [[array([11, 13, 51]) array([17])]
  [array([13, 51, 17]) array([18])]
  [array([51, 17, 18]) array([17])]
  [array([17, 18, 17]) array([10])]]]

We could reshape that to a (8,2) array (still object dtype):

In [14]: print(Out[11].reshape(-1,2))
[[array([1, 3, 5]) array([7])]
 [array([3, 5, 7]) array([8])]
 [array([5, 7, 8]) array([7])]
 [array([7, 8, 7]) array([1])]
 [array([11, 13, 51]) array([17])]
 [array([13, 51, 17]) array([18])]
 [array([51, 17, 18]) array([17])]
 [array([17, 18, 17]) array([10])]]

Since the inner most arrays have a mixed size - some 3 some 1, the result can only be object dtype - or list of lists. That's isn't optimal for array calculations.

Commas are part of the display, along with [] and words like array. Together they give us clues as to the underlying objects, whether they are lists or arrays. Equally important are the shape and dtype (if the object is an array) or length if a list.

===

Another answer uses a striding_tricks function. Here's that method in more detail. While x is a view, slicing and reshaping will make copies, so it's hard to say whether this is any faster. For this small example I bet your code is faster, but for larger case it might not be.

In [16]: np.lib.stride_tricks.sliding_window_view(a,(1,4))
Out[16]: 
array([[[[ 1,  3,  5,  7]],

        [[ 3,  5,  7,  8]],

        [[ 5,  7,  8,  7]],

        [[ 7,  8,  7,  1]]],


       [[[11, 13, 51, 17]],

        [[13, 51, 17, 18]],

        [[51, 17, 18, 17]],

        [[17, 18, 17, 10]]]])
In [17]: x = np.lib.stride_tricks.sliding_window_view(a,(1,4))
In [18]: x.shape
Out[18]: (2, 4, 1, 4)

That's a 4d view of the original 1d array.

Your size 3 'arrays' can be sliced from that:

In [19]: x[:,:,0,:3]
Out[19]: 
array([[[ 1,  3,  5],
        [ 3,  5,  7],
        [ 5,  7,  8],
        [ 7,  8,  7]],

       [[11, 13, 51],
        [13, 51, 17],
        [51, 17, 18],
        [17, 18, 17]]])
In [20]: x[:,:,0,:3].reshape(-1,3)
Out[20]: 
array([[ 1,  3,  5],
       [ 3,  5,  7],
       [ 5,  7,  8],
       [ 7,  8,  7],
       [11, 13, 51],
       [13, 51, 17],
       [51, 17, 18],
       [17, 18, 17]])

and the 1 element column:

In [21]: x[:,:,0,-1].reshape(-1,1)
Out[21]: 
array([[ 7],
       [ 8],
       [ 7],
       [ 1],
       [17],
       [18],
       [17],
       [10]])

These 2 arrays may be more useful than your object out.

The arrays shown in [14] could be split into 2 similar arrays:

In [27]: np.stack(arr.reshape(-1,2)[:,0])
Out[27]: 
array([[ 1,  3,  5],
       [ 3,  5,  7],
       [ 5,  7,  8],
       [ 7,  8,  7],
       [11, 13, 51],
       [13, 51, 17],
       [51, 17, 18],
       [17, 18, 17]])
In [28]: arr.reshape(-1,2)[:,1].astype(int)
Out[28]: array([ 7,  8,  7,  1, 17, 18, 17, 10])

Answered By - hpaulj