[FIXED] How can I read hdf5 files stored as 1-D array. and view them as images?

Issue

I have a large image classification dataset stored in the format .hdf5. The dataset has the labels and the images stored in the .hdf5 file. I am unable to view the images as they are store in form of an array. The dataset reading code that I have used is as follows,

import h5py
import numpy
f = h5py.File('data/images.hdf5', 'r')
print(list(f.keys()))

['datasets']

group = f['datasets']
list(group.keys())

['car']

Now when I read the group cars I have the following output,

data = group['car']
data.shape,data[0].shape,data[1].shape

((51,), (383275,), (257120,)

So it looks like there are 51 images for label car and images are stored as 383275 and 257120 dimensional arrays, with no information about their height and width dimensions. I want to save the images as RGB again. Next following the code here, I tried to read the images.

import numpy as np
from PIL import Image

# hdf = h5py.File("Sample.h5",'r')
array = data[0]
img = Image.fromarray(array.astype('uint8'), 'RGB')
img.save("yourimage.thumbnail", "JPEG")
img.show()

Unfortunately, the following error is received.

File /usr/local/lib/python3.8/dist-packages/PIL/Image.py:784, in Image.frombytes(self, data, decoder_name, *args)
    781 s = d.decode(data)
    783 if s[0] >= 0:
--> 784     raise ValueError("not enough image data")
    785 if s[1] != 0:
    786     raise ValueError("cannot decode image data")

ValueError: not enough image data

References I have already checked the hdf group help library etc. Any help will be highly useful. Thanks.

Solution

First, the f['datasets']['car'] object is a DATASET, not a GROUP. Second, based this output, I think your dataset is a dataset with rows of variable length arrays (aka "ragged" array).

# this is group object reference:
group = f['datasets']
# these are equivalent dataset object references:
data = group['car']  
data = f['datasets']['car']  
# this gives the dataset shape (# of rows), then the shape for data on row 0 and  row 1:
data.shape,data[0].shape,data[1].shape
((51,), (383275,), (257120,)

From this, I think you have 51 rows, each with a single 1-D array of different sizes. If I'm correct, the original image array was flattened.

To reconstruct each image you will need the original shape of the image. Hopefully it's documented somewhere in the file. Do you have the schema definition for this file? If not, you will have to determine that somehow. [If so, someone didn't do you any favors. This is an example of something "clever", but not very helpful for future users.]

Hopefully they left you some "bread crumbs" to get each image size. As @Manoj Bhosle suggests, there might be in another dataset. Or, they could be saved as attributes. Either way, you will have to interrogate the file to figure this out. The easiest way is to open it with HDFView and examine it. HDFView can be downloaded from The HDF Group.

You can access any attributes on the 'car' dataset with this code:

with h5py.File('data/images.hdf5') as h5f:
    ds = h5f['datasets']['car']
    for k in ds.attrs.keys():
        print(f"{k} => {ds.attrs[k]}")

You can check the size of the array stored on each row in the 'car' dataset with this code:

with h5py.File('data/images.hdf5') as h5f:
    ds = h5f['datasets']['car']
    for row in ds:
        print(row.shape, row[0].shape)

Answered By - kcw78