[FIXED] how to use is_valid_file of torchvision.datasets.ImageFolder

Issue

I am trying to load some images to train a small model. However, I couldn't use is_valid_file to distinguish my train and test images. For example my file is like that :

vegetables/tomato/gallery.jpg

for train set and

vegetables/tomato/probe.jpg

for test set. Basically every file like tomato and other selected vegetables has 2 images both test and train image.

Question: How can I use is_valid_file to distinguish them. I checked the documentation but I didn't understand.

This is my code

trainset = torchvision.datasets.ImageFolder(
      root = "vegetables",
      transform=imagenet_transform
    )

testset = torchvision.datasets.ImageFolder(
  root = "vegetables",
  transform=imagenet_transform
)

Solution

I had the problem too!!! Only not to the vegetable problem, but to the cat-fish problem. You can find the datasets here on this page (https://www.kaggle.com/datasets/zuraiz/fish-vs-cats-imagenet-subdataset). The downloaded datasets have to be saved in the same folder where you saved your .ipynb file. I use Google Colab (makes things a bit easier), which means I have an .ipynb file.

import torchvision
from PIL import Image, ImageFile
from torchvision import transforms
from torchvision.datasets import ImageFolder 

from google.colab import drive
drive.mount('/content/drive')

ImageFile.LOAD_TRUNCATED_IMAGES = False

def check_Image(path):
  try:
    im = Image.open(path)
    return True
  except:
    return False
# train_data_path = "/.train/"

    

in order to get your path you have to go (in my case) - drive -> MyDrive -> Pytorch Projects -> 02 Chapter -> Fish-vs-Cats -> train -> 3 points on the right -> copy path

train_data_path = "/content/drive/MyDrive/Pytorch Projects/02 Chapter/Fish-vs-Cats/train"



img_transform = transforms.Compose([
    transforms.Resize((64,64)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225])
])
   
train_data = ImageFolder(root=train_data_path, 
                         transform=img_transform, 
                         is_valid_file=check_Image)

Answered By - CGD 16