[FIXED] How to convert list of dictionaries to NumPy recarray (access fields by name)

Issue

I notice it is possible to access a variable in a Nd Array items Container using dot notation.

This is especially true for a file produced by the loadmat package of scipy.

For example,the following is a variable from the Nd Array items Container, that accessed using the dot notation.

dot_notation_output=stru[0].fieldA

Im curios how to reproduce something similar for a given nested dict as below.

struc=[{'fieldA': 11.02, 'fieldB': 2.69,'fieldC': 2.69}, {"fieldA": 21.4, "fieldB": 66.69,'fieldC': 2.69},
           {"fieldA": 100,"fieldB": 200,'fieldC': 2.69}]

Creating directly from np.array does not reproduce the finding above

np.array(struc)

The struc from Matlab was created as follow

for idx = 1:3
    stru(idx) = create_Structure();
    stru(idx).fieldA = '1';
    stru(idx).fieldB = 3;
    stru(idx).fieldC = 44;

end

save('struc_mat.mat','stru')
    function s = create_Structure()
        %% Create a structure 
        s = struct( ...
            'fieldA', NaN,'fieldB', NaN,'fieldC',NaN);
    end

The struc produced in Matlab, can be open in Python with the scipy-loadmat

from scipy.io import loadmat
stru = loadmat ( 'struc_mat.mat', squeeze_me=True, struct_as_record=False )
stru = stru['stru']
dot_notation_output=stru[0].fieldA

Solution

Once you know how to define a record array, the process to create one from your dictionary is relatively straightforward.

• First you have to get the dictionary keys (to use as field names).
• Use these to define the "dtype" for the recarray. (Note: You also need to define each fields "dtype". I assumed floats. You can add logic to check the dictionary value types to ensure an appropriate type is used.)
• Use the dtype to create an empty recarray.
• Finally, loop thru the dictionary (again) to populate the array based on field names and list position (used as array index).

Note, you can reference array values in 2 ways: 1) with dot notation you described: recarr[0].fieldA, or 2) with the name as an array index: recarr[0]['fieldA']. I prefer the second method as it gives a programtic way to access the values when the field name is a variable, and not hard-coded.

Code to create a recarray with your data below:

import numpy as np

struc=[{'fieldA': 11.02,'fieldB': 2.69, 'fieldC': 2.69}, 
       {'fieldA': 21.4, 'fieldB': 66.69,'fieldC': 2.69},
       {'fieldA': 100,  'fieldB': 200,  'fieldC': 2.69}]

keys = []
for d in struc:
    for k in d.keys():
        if k not in keys:
            keys.append(k)

dt = np.dtype([ (name,float) for name in keys ])
recarr = np.recarray((len(struc),),dtype=dt)
print(recarr.dtype)    

for i, d in enumerate(struc):
    for key,val in d.items():
        recarr[i][key] = val

print(recarr)

    

Answered By - kcw78