[FIXED] pytorch dataloader stucked if using opencv resize method

Issue

I can run all the cells of the tutorial notebook of Pytorch about dataloading (pytorch tutorial). But when I use OpenCV in place of Skimage to resize the image, the dataloader gets stuck, i.e nothing happens.

In the Rescale class:

class Rescale(object):
    .....
    def __call__(self, sample):
       ....
       #img = transform.resize(image, (new_h, new_w))
       img = cv2.resize(image, (new_h, new_w))
       .....

The dataloader and the for loop are defined with:

dataloader = DataLoader(transformed_dataset, batch_size=4,
                        shuffle=True, num_workers=4)

for i_batch, sample_batched in enumerate(dataloader):
    print(i_batch, sample_batched['image'].size(),
          sample_batched['landmarks'].size())

I can get the iterator to print something if num_workers=0. It looks like opencv does not play well with the Multiprocessing of pytorch. I would really prefer to use same package to transform the images at train time and test time (and I am already using OpenCV for the image rescale at test time). Any suggestions would be greatly appreciated.

Solution

I had a very similar problem and that's how I solved it:

when you import cv2 set cv2.setNumThreads(0) and then you can set num_workers>0 in the dataloader in PyTorch.

Seems like OpenCV tries to multithread and somewhere something goes into a deadlock.

Hope it helps.

Answered By - Aidas Liaudanskas