[FIXED] How import a 3D numpy array?

Issue

Using python 3, I am trying to process a set of data in a four-column text file: The first column is the x index, the second column is the y index and the third column is the z index, or depth index. The fourth column is the data value. The values in the text file look like this:

0 0 0 0.0
1 0 0 0.0
2 0 0 2.0
0 1 0 0.0
1 1 0 0.0
2 1 0 2.0
0 2 0 0.0
1 2 0 0.0
2 2 0 2.0
0 0 1 0.0
1 0 1 0.0
2 0 1 2.0
0 1 1 0.0
1 1 1 0.0
2 1 1 2.0
0 2 1 0.0
1 2 1 0.0
2 2 1 2.0

Is there a way to construct a 3D numpy array with shape (2,3,3)?

[[[0 0 0]
  [0 0 0]
  [2 2 2]],
 [[0 0 0]
  [0 0 0]
  [2 2 2]]]

While this example shows 18 rows wanting to be shaped into a (2,3,3) array, my actual data is 512x512x49 (12845056) rows and I'd like to shape them into a (512,512,49) array. If the solution could efficiently parse a greater number of rows, that would be appreciated, but I understand python has some fundamental speed limitations.

This is what I have tried so far:

import numpy as np
f = "file_path.txt"
data = np.loadtxt(f)
data = data.reshape((512,512,49))

but this gives the following error:

ValueError: cannot reshape array of size 51380224 into shape (512,512,49)

I was surprised by this error since 51380224 is not equal to the number of rows in my loaded array (12845056). Also, I suspect numpy needs information that the first, second, and third columns are not values, but indices along which to shape the values in the fourth column. I am not sure how to achieve this, and am open to solutions in either numpy or pandas.

Solution

With your sample file:

In [94]: txt = """0 0 0 0.0
    ...: 1 0 0 0.0
    ...: 2 0 0 2.0
    ...: 0 1 0 0.0
    ...: 1 1 0 0.0
    ...: 2 1 0 2.0
    ...: 0 2 0 0.0
    ...: 1 2 0 0.0
    ...: 2 2 0 2.0
    ...: 0 0 1 0.0
    ...: 1 0 1 0.0
    ...: 2 0 1 2.0
    ...: 0 1 1 0.0
    ...: 1 1 1 0.0
    ...: 2 1 1 2.0
    ...: 0 2 1 0.0
    ...: 1 2 1 0.0
    ...: 2 2 1 2.0""".splitlines()

The straight forward load detects 4 columns, and makes all values float:

In [95]: data = np.genfromtxt(txt)
In [96]: data.shape
Out[96]: (18, 4)

We could work from those, converting the float indices to integer. Or we can load the file in 2 steps:

In [103]: indices = np.genfromtxt(txt, usecols=[0,1,2], dtype=int)
In [104]: values = np.genfromtxt(txt, usecols=[3])

and use those values to fill in an array:

In [105]: res = np.zeros((2,3,3),float)
In [107]: res[indices[:,2],indices[:,0],indices[:,1]] = values
In [108]: res
Out[108]: 
array([[[0., 0., 0.],
        [0., 0., 0.],
        [2., 2., 2.]],

       [[0., 0., 0.],
        [0., 0., 0.],
        [2., 2., 2.]]])

Answered By - hpaulj