[FIXED] genfromtxt read data of different types as array or arrays

Issue

I am trying to import data from a text file with a varying number of columns and insert it into an array of arrays. I know that the first column will always be a string and the next three columns will be integers, but so far I have only managed to read the file as an array of tuples

i have tried using dtype=(object,int,int,int)

from io import StringIO
import numpy as np

new_string = StringIO("01/23/2020, 32, 0, 2 \n01/31/2020' ,436 ,0 ,10")
new_result = np.genfromtxt(new_string, dtype=(object,int,int,int), encoding="unicode"
                           , delimiter=",")

print("File data:",new_result )

output:

File data: [('01/23/2020',  32, 0,  2) ("01/31/2020' ", 436, 0, 10)]

I want the output tolook like this

[['01/23/2020' 32 0 2]
 ['01/31/2020'  436 0 10]]

to that

new_result == np.array( [['01/23/2020',32,0,2],
                         ['01/31/2020', 436, 0, 10]],dtype=object)

will be True

Solution

Specifying a dtype like that produces a structured array https://numpy.org/doc/stable/user/basics.rec.html

In [40]: new_string = StringIO("01/23/2020, 32, 0, 2 \n01/31/2020' ,436 ,0 ,10")
    ...: new_result = np.genfromtxt(new_string, dtype=(object,int,int,int), encoding="unicode"
    ...:                            , delimiter=",")

This is a 1d array, with a compound dtype. The print display just shows the elements, or records, as tuples, but the repr display shows the dtype as well:

In [41]: new_result
Out[41]: 
array([(b'01/23/2020',  32, 0,  2), (b"01/31/2020' ", 436, 0, 10)],
      dtype=[('f0', 'O'), ('f1', '<i4'), ('f2', '<i4'), ('f3', '<i4')])

In [42]: new_result.dtype
Out[42]: dtype([('f0', 'O'), ('f1', '<i4'), ('f2', '<i4'), ('f3', '<i4')])

Fields are accessed by name:

In [43]: new_result['f0']
Out[43]: array([b'01/23/2020', b"01/31/2020' "], dtype=object)

In [44]: new_result['f1']
Out[44]: array([ 32, 436])

The main structured array doc page suggests using a recfunctions function to convert dtypes:

In [46]: import numpy.lib.recfunctions as rf

Unfortunately the object field is giving that problems:

In [48]: arr = rf.structured_to_unstructured(new_result, dtype=object)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [48], in <cell line: 1>()
----> 1 arr = rf.structured_to_unstructured(new_result, dtype=object)

File <__array_function__ internals>:5, in structured_to_unstructured(*args, **kwargs)

File ~\anaconda3\lib\site-packages\numpy\lib\recfunctions.py:980, in structured_to_unstructured(arr, dtype, copy, casting)
    978 with suppress_warnings() as sup:  # until 1.16 (gh-12447)
    979     sup.filter(FutureWarning, "Numpy has detected")
--> 980     arr = arr.view(flattened_fields)
    982 # next cast to a packed format with all fields converted to new dtype
    983 packed_fields = np.dtype({'names': names,
    984                           'formats': [(out_dtype, dt.shape) for dt in dts]})

File ~\anaconda3\lib\site-packages\numpy\core\_internal.py:494, in _view_is_safe(oldtype, newtype)
    491     return
    493 if newtype.hasobject or oldtype.hasobject:
--> 494     raise TypeError("Cannot change data-type for object array.")
    495 return

Let's try the dtype=None option (and clean up the string a bit):

In [49]: new_string = StringIO("01/23/2020, 32, 0, 2 \n01/31/2020 ,436 ,0 ,10")
    ...: new_result = np.genfromtxt(new_string, dtype=None, encoding="unicode"
    ...:                            , delimiter=",")

In [50]: new_result
Out[50]: 
array([('01/23/2020',  32, 0,  2), ('01/31/2020 ', 436, 0, 10)],
      dtype=[('f0', '<U11'), ('f1', '<i4'), ('f2', '<i4'), ('f3', '<i4')])

Same as your case except the string dtype field.

But that doesn't help; it must be the target dtype that the function doesn't like (or both):

In [51]: arr = rf.structured_to_unstructured(new_result, dtype=object)
...
TypeError: Cannot change data-type for object array.

But we can convert the numeric fields, producing a 2d int array:

In [52]: arr = rf.structured_to_unstructured(new_result[['f1','f2','f3']], dtype=int)

In [53]: arr
Out[53]: 
array([[ 32,   0,   2],
       [436,   0,  10]])

Assigning fields to object array

In [65]: new_string = "01/23/2020, 32, 0, 2 \n01/31/2020, 436 ,0 ,10".splitlines()
    ...: new_result = np.genfromtxt(new_string, dtype='O,i,i,i', encoding="unicode"
    ...:                            , delimiter=",")

In [66]: new_result
Out[66]: 
array([(b'01/23/2020',  32, 0,  2), (b'01/31/2020', 436, 0, 10)],
      dtype=[('f0', 'O'), ('f1', '<i4'), ('f2', '<i4'), ('f3', '<i4')])

Create a target array:

In [67]: arr = np.empty((2,4),object)    
In [68]: for i,f in enumerate(new_result.dtype.fields):
    ...:     arr[:,i] = new_result[f]
    ...:     

In [69]: arr
Out[69]: 
array([[b'01/23/2020', 32, 0, 2],
       [b'01/31/2020', 436, 0, 10]], dtype=object)

Many of the recfunctions do something like this - create a target array, and copy data by field name. Usually a structured array has many more records than fields, so this iteration by field is relatively efficient.

unpack

If you specify unpack, the result is separate arrays for each column/field

In [74]: new_string = "01/23/2020, 32, 0, 2 \n01/31/2020, 436 ,0 ,10".splitlines()
    ...: new_result = np.genfromtxt(new_string, dtype='O,i,i,i', unpack=True
    ...:                            , delimiter=",")

In [75]: new_result
Out[75]: 
[array([b'01/23/2020', b'01/31/2020'], dtype=object),
 array([ 32, 436], dtype=int32),
 array([0, 0], dtype=int32),
 array([ 2, 10], dtype=int32)]

They can then be concatenated with stack:

In [77]: np.stack(new_result, axis=1)
Out[77]: 
array([[b'01/23/2020', 32, 0, 2],
       [b'01/31/2020', 436, 0, 10]], dtype=object)

Answered By - hpaulj